Unbootable system: systemd is gone

Today my client sent me this screen on their computer:

This means that systemd is gone. Same as it happened just a week ago with glibc.

I suspect the culprit to be on manjaro-system, which uses extensively:

pacman -Rdd

When the Arch wiki explicitly states:

Avoid using the -d option with pacman. pacman -Rdd package skips dependency checks during package removal. As a result, a package providing a critical dependency could be removed, resulting in a broken system.

My client’s computer has a timer that from time to time removes orphans using:

pacman --noconfirm --remove --recursive --unneeded $(pacman --query --deps --unrequired --quiet || true)

So the most probable answer is that manjaro-system, in some rare circumstances, is messing deps. And along with the timer critical components get removed.

Hence if I was you I would have a look, and maybe consider a safer alternative. Otherwise, sadly, I may be forced to remove manjaro-system.

I haven’t yet retrieved the pacman.log from his system. The last time this happened the log was:

[2023-01-24T10:57:08+0100] [PACMAN] Running ‘pacman --sync --refresh --sysupgrade --noconfirm’
[2023-01-24T10:57:08+0100] [ALPM] transaction started
[2023-01-24T10:57:08+0100] [ALPM] upgraded manjaro-system (20220727-1 → 20221227-1)
[2023-01-24T10:57:08+0100] [ALPM-SCRIPTLET] [1;1m [1;32m==> [1;0m [1;1m Checking for ‘os-prober’ setup . … [1;0m
[2023-01-24T10:57:08+0100] [ALPM-SCRIPTLET] We will reenable ‘os-prober’ for you …
[2023-01-24T10:57:09+0100] [ALPM-SCRIPTLET] Generating a grub configuration file…
[2023-01-24T10:57:09+0100] [ALPM-SCRIPTLET] Found linux image: /boot/vmlinuz-linux-zen
[2023-01-24T10:57:09+0100] [ALPM-SCRIPTLET] Initial memory image found: /boot/intel-ucode.img /boot/initramfs-linux-zen.img
[2023-01-24T10:57:09+0100] [ALPM-SCRIPTLET] Found initrd fallback image: /boot/initramfs-linux-zen-fallback.img
[2023-01-24T10:57:09+0100] [ALPM-SCRIPTLET] Warning: os-prober will be executed to detect other bootable partitions.
[2023-01-24T10:57:09+0100] [ALPM-SCRIPTLET] Its output will be used to detect bootable binaries on them and create new boot entries.
[2023-01-24T10:57:10+0100] [ALPM-SCRIPTLET] Found memtest86+ image: /boot/memtest86+/memtest.bin
[2023-01-24T10:57:10+0100] [ALPM-SCRIPTLET] done
[2023-01-24T10:57:10+0100] [ALPM] transaction completed
[2023-01-24T10:57:10+0100] [ALPM] running ‘30-systemd-update.hook’…
[2023-01-24T10:57:10+0100] [PACMAN] starting full system upgrade
[2023-01-24T10:57:37+0100] [ALPM] transaction started
[2023-01-24T10:57:37+0100] [ALPM] transaction failed

/sbin/init is a link to /lib/systemd/systemd

The link is provided by systemd-sysvcompat

/sbin/init not being found is not the same as systemd being gone

the link may be gone - for whatever reason

If you could inspect the system (chroot or just mount) then you’d know.

If anything, systemd-sysvcompat may be gone and the link with it.

Investigate this timer - where does it come from?
Although I dont think that:
pacman --query --deps --unrequired
could ever come up with anything like systemd or systemd-sysvcompat

… and linux-zen is not easy to get on a Manjaro system

That system has had to be … modified.

2 Likes

I will on Monday in two weeks, because he has gone on holiday.

I will inspect if that package is still installed, and if any other package is orphan or something alike.

I coded it.

Except maybe if the package has been orphaned.

It’s just the Zen kernel as compiled in Arch, back-ported through a personal repository.

It has that personal repo I packaged myself.

It has a timer that keeps the system upgraded and free of orphans. It has worked for over a year without issues for a bunch of people, and hasn’t been modified in a while.

Omitting checks, what it runs is just this:

//refresh databases once
pacman --sync --refresh

//update keyrings
pacman --sync --noconfirm --downloadonly archlinux-keyring
pacman --sync --noconfirm archlinux-keyring
pacman --sync --noconfirm --downloadonly manjaro-keyring
pacman --sync --noconfirm manjaro-keyring

//update system
pacman --sync --sysupgrade --noconfirm --downloadonly
pacman --sync --sysupgrade --noconfirm

//prune orphans
orphans=“$(pacman --query --deps --unrequired --quiet || true)”
pacman --noconfirm --remove --recursive --unneeded “${orphans}”

While it is installing the packages, and not just downloading them, it cannot be interrupted by power off. systemd will wait for it indefinitely to finish.

The point I tried to make - and I think you got that - is
that this:

is very unlikely to be the actual “culprit”
for this malfunction.

We will see … :slightly_smiling_face:

3 Likes

The use of pacman -Rdd is not a problem when it is used correctly.

1 Like

Is that message before or after the system transitions from the initramfs to the actual system :thinking:
Looks to me like it is still in the initrd phase…

Would be nice to know the kernel boot parameters used by the bootloader…

1 Like

Fact - You are blaming a Manjaro maintenance script for the mishap at your client.

When looking at the change history for the maintenance script https://gitlab.manjaro.org/packages/core/manjaro-system/-/commits/master/manjaro-update-system.sh it is clear that the script is not used very often and when used it is targeting specific issues which would otherwise require manual intervention.

The last issue the script handled was the deprecation of dbus-x11 and the use of -Rdd ensures that no other packages are touched - except the target dbus-x11 which immediately is replaced with dbus package.

The warning in the Arch Wiki is a well placed warning - which should be heeded by the casual user.

The reason for having these advanced possibilities with pacman - is to be able to replace important packages - packages which - should they be removed using standard commands - may require a cascading removal.

Therefore - a concent removal using -Rdd dbus-x11 followed by an immediate replacement -S dbus - is the better decision.

2 Likes

I will have a look.

Still that won’t explain how some files disappeared previously.

Between transactions the missing dbus may have caused the failure.

For example a hook or command may be triggered in between, but that hook or command requires dbus itself.

That’s the reason why installations are done first in a chroot, so no files are missing between transactions.

For instance systemd requires dbus, and its hook is just triggered by writing into “/etc”:

[2023-01-24T10:57:10+0100] [ALPM] running ‘30-systemd-update.hook’…
[2023-01-24T10:57:10+0100] [PACMAN] starting full system upgrade
[2023-01-24T10:57:37+0100] [ALPM] transaction started
[2023-01-24T10:57:37+0100] [ALPM] transaction failed

[Trigger]
Type = Path
Operation = Install
Operation = Upgrade
Operation = Remove
Target = usr/

[Action]
Description = Arming ConditionNeedsUpdate…
When = PostTransaction
Exec = /usr/share/libalpm/scripts/systemd-hook update

So maybe that’s also a possibility.

Probably the only safe way of transitioning from one package to another is by using “replaces” on its PKGBUILD.

A phrase often used is

Logs - otherwise it didn’t happpen …

This goes for your issue as well - a dumb screenshot is not proving anything but a failure - there is no evidence of what caused the failure …

That is impossible - and no matter what you think - your client issue was after a restart of the system - which would infer a previously functional system - which would not have been the case if the complete systemd package was missing.

Furthermore you are grasping straws when referencing hooks … just check the dbus package …

The reload hook may fail upon removal - but as the package is immediately replaced and the subsequent reload hook will execute as expected …

This still doesn’t encounter for your client failure after a restart …

You should examine your clients pacman log closely - as I think you may find the answer there.

I would be investigating what the scripts of your zenned linux actually does … because right now you are blaming a distribution - which as this very moment does not exhibit any issues to a wrongly removed systemd package … think about it …

I am a developer … developing real world code… where my coding immediately affects dozens of scandinavian salesreps … I know … the first place I look when an error is reported … is my code … think about it …

Not all of my code is publicly available … some is … fhdk (Frede H) · GitHub

1 Like

I’m not repeating myself and you will not receive another warning. Please stop wasting our time with wild, inaccurate, baseless accusations.