The [Stable Update] 2021-06-14 prevent me from booting my system

I updated using pamac for the first time (instead of pacman). I restarted after the update and now I cannot log into my computer. When I boot I get the message:

[FAILED] Failed to start Rule-based Manager for Device Events and Files
[FAILED] Failed to start Rule-based Manager for Device Events and Files
[FAILED] Failed to start Rule-based Manager for Device Events and Files
[FAILED] Failed to start Rule-based Manager for Device Events and Files
[FAILED] Failed to start Rule-based Manager for Device Events and Files
[FAILED] Failed to start Rule-based Manager for Device Events and Files
[FAILED] Failed to start Entropy Daemon based on the HAVEGE algorithm.
[TIME] Timed out waiting for device /dev/disk/by-uuid and a bunch of numbers
[DEPEND] Dependency failed for /dev/disk/by-uuid numbers as above
[DEPEND] Dependency failed for Swap.
[TIME] Timed out waiting for device /dev/disk/by-uuid and another set of numbers
[DEPEND] Dependency failed for /boot/efi
[DEPEND] Dependency failed for Local File Systems
[DEPEND] Dependency failed for File System Check on /dev/disk/by-uuid and the second set of numbers
[FAILED] Failed to start Network Time Synchronization
[FAILED] Failed to start Network Time Synchronization
[FAILED] Failed to start Network Time Synchronization
[FAILED] Failed to start Network Time Synchronization
[FAILED] Failed to start Network Time Synchronization
[FAILED] Failed to start Network Time Synchronization
You are in emergency mode. After logging in, type "journalctl -xb" to view 
system logs, "systemctl reboot" to reboot, "systemctl default" or "exit"
to boot into default mode.
Give root password for maintenance
(or press Control-D to continue):

My password does not work, nor does pressing Control-D do anything. I really have no idea how to go about fixing this and I am a bit worried that I am going to be locked out of my computer for good. Can anyone help me with this?

This issue can be several problems. You’re not locked out of your PC. Do you have the installation media still available? If not, get a new ISO and flash it on a USB stick. Then boot up the system and chroot into your installed system.

Check the following things:

  • post the file /var/log/pacman.log so we see what got updated
  • try to check if the update had completed: either sudo pacman -Syu or pamac update
  • install a newer kernel like linux510 via sudo mhwd-kernel -i linux510. Note: Kernel 4.19 seems not to boot with Systemd 248 series.

See also here:

2 Likes

Seems the issue is with systemd 248 and Kernel 4.19 series. I just tried to boot and got the same lines reported. However, I was not able to boot further to a secure terminal. So double check if you’re running that kernel.

For example I get these error messages:

Jun 14 20:40:35 development systemd[422]: haveged.service: Failed at step NAMESPACE spawning /usr/bin/haveged: Invalid argument
Jun 14 20:40:35 development systemd[1]: Finished Load AppArmor profiles managed internally by snapd.
Jun 14 20:40:36 development systemd-modules-load[362]: Inserted module 'nvidia'
Jun 14 20:40:36 development systemd[1]: haveged.service: Main process exited, code=exited, status=226/NAMESPACE
Jun 14 20:40:36 development systemd[1]: haveged.service: Failed with result 'exit-code'.
Jun 14 20:40:37 development systemd[1]: haveged.service: Scheduled restart job, restart counter is at 3.
Jun 14 20:40:37 development systemd[1]: Stopped Entropy Daemon based on the HAVEGE algorithm.
Jun 14 20:40:37 development systemd[1]: Started Entropy Daemon based on the HAVEGE algorithm.
Jun 14 20:40:37 development systemd[431]: haveged.service: Failed to set up mount namespacing: /run/systemd/unit-root/dev: Invalid argument
Jun 14 20:40:37 development systemd[431]: haveged.service: Failed at step NAMESPACE spawning /usr/bin/haveged: Invalid argument
Jun 14 20:40:37 development systemd[1]: haveged.service: Main process exited, code=exited, status=226/NAMESPACE
Jun 14 20:40:37 development systemd[1]: haveged.service: Failed with result 'exit-code'.
Jun 14 20:40:37 development systemd[1]: haveged.service: Scheduled restart job, restart counter is at 4.
Jun 14 20:40:37 development systemd[1]: Stopped Entropy Daemon based on the HAVEGE algorithm.
Jun 14 20:40:37 development systemd[1]: Started Entropy Daemon based on the HAVEGE algorithm.
Jun 14 20:40:38 development systemd[433]: haveged.service: Failed to set up mount namespacing: /run/systemd/unit-root/dev: Invalid argument
Jun 14 20:40:38 development systemd[433]: haveged.service: Failed at step NAMESPACE spawning /usr/bin/haveged: Invalid argument
Jun 14 20:40:38 development systemd[1]: haveged.service: Main process exited, code=exited, status=226/NAMESPACE
Jun 14 20:40:38 development systemd[1]: haveged.service: Failed with result 'exit-code'.
Jun 14 20:40:38 development systemd[1]: haveged.service: Scheduled restart job, restart counter is at 5.
Jun 14 20:40:38 development systemd[1]: Stopped Entropy Daemon based on the HAVEGE algorithm.
Jun 14 20:40:38 development systemd[1]: haveged.service: Start request repeated too quickly.
Jun 14 20:40:38 development systemd[1]: haveged.service: Failed with result 'exit-code'.
Jun 14 20:40:38 development systemd[1]: Failed to start Entropy Daemon based on the HAVEGE algorithm.

And these

Jun 14 20:40:33 development systemd[1]: systemd-udevd.service: Main process exited, code=exited, status=226/NAMESPACE
Jun 14 20:40:33 development systemd[1]: systemd-udevd.service: Failed with result 'exit-code'.
Jun 14 20:40:33 development systemd[1]: Failed to start Rule-based Manager for Device Events and Files.
Jun 14 20:40:33 development systemd[1]: systemd-udevd.service: Scheduled restart job, restart counter is at 2.
Jun 14 20:40:33 development systemd[1]: Stopped Rule-based Manager for Device Events and Files.
Jun 14 20:40:33 development systemd[1]: Starting Rule-based Manager for Device Events and Files...
Jun 14 20:40:33 development systemd[392]: systemd-udevd.service: Failed to set up mount namespacing: /run/systemd/unit-root/proc/sys/kernel/hostname: Device or resource busy
Jun 14 20:40:33 development systemd[392]: systemd-udevd.service: Failed at step NAMESPACE spawning /usr/lib/systemd/systemd-udevd: Device or resource busy
Jun 14 20:40:33 development systemd[1]: systemd-udevd.service: Main process exited, code=exited, status=226/NAMESPACE
Jun 14 20:40:33 development systemd[1]: systemd-udevd.service: Failed with result 'exit-code'.
Jun 14 20:40:33 development systemd[1]: Failed to start Rule-based Manager for Device Events and Files.
Jun 14 20:40:33 development systemd[1]: systemd-udevd.service: Scheduled restart job, restart counter is at 3.
Jun 14 20:40:40 development ufw-init[441]: Skip starting firewall: ufw (not enabled)
Jun 14 20:40:33 development systemd[1]: Stopped Rule-based Manager for Device Events and Files.
Jun 14 20:40:33 development systemd[1]: Starting Rule-based Manager for Device Events and Files...
Jun 14 20:40:33 development systemd[393]: systemd-udevd.service: Failed to set up mount namespacing: /run/systemd/unit-root/proc/sys/kernel/hostname: Device or resource busy
Jun 14 20:40:33 development systemd[393]: systemd-udevd.service: Failed at step NAMESPACE spawning /usr/lib/systemd/systemd-udevd: Device or resource busy
Jun 14 20:40:33 development systemd[1]: systemd-udevd.service: Main process exited, code=exited, status=226/NAMESPACE
Jun 14 20:40:33 development systemd[1]: systemd-udevd.service: Failed with result 'exit-code'.
Jun 14 20:40:33 development systemd[1]: Failed to start Rule-based Manager for Device Events and Files.
Jun 14 20:40:33 development systemd[1]: systemd-udevd.service: Scheduled restart job, restart counter is at 4.
Jun 14 20:40:33 development systemd[1]: Stopped Rule-based Manager for Device Events and Files.
Jun 14 20:40:33 development systemd[1]: Starting Rule-based Manager for Device Events and Files...
Jun 14 20:40:33 development systemd[394]: systemd-udevd.service: Failed to set up mount namespacing: /run/systemd/unit-root/proc/sys/kernel/hostname: Device or resource busy
Jun 14 20:40:33 development systemd[394]: systemd-udevd.service: Failed at step NAMESPACE spawning /usr/lib/systemd/systemd-udevd: Device or resource busy
Jun 14 20:40:33 development systemd[1]: systemd-udevd.service: Main process exited, code=exited, status=226/NAMESPACE
Jun 14 20:40:33 development systemd[1]: systemd-udevd.service: Failed with result 'exit-code'.
Jun 14 20:40:33 development systemd[1]: Failed to start Rule-based Manager for Device Events and Files.
Jun 14 20:40:33 development systemd[1]: systemd-udevd.service: Scheduled restart job, restart counter is at 5.
Jun 14 20:40:33 development systemd[1]: Stopped Rule-based Manager for Device Events and Files.
Jun 14 20:40:33 development systemd[1]: systemd-udevd.service: Start request repeated too quickly.
Jun 14 20:40:33 development systemd[1]: systemd-udevd.service: Failed with result 'exit-code'.
Jun 14 20:40:33 development systemd[1]: Failed to start Rule-based Manager for Device Events and Files.
Jun 14 20:40:33 development systemd[1]: systemd-udevd-control.socket: Failed with result 'service-start-limit-hit'.
Jun 14 20:40:33 development systemd[1]: systemd-udevd-kernel.socket: Failed with result 'service-start-limit-hit'.

I am using that kernel. I think this is what you asked me for earlier:

cat /var/log/pacman.log
[2021-06-14T20:10:08+0000] [PACMAN] Running ‘pacman --noconfirm --cachedir /var/cache/pacman/pkg --config /opt/mhwd/pacman-mhwd.conf --root / --needed -Sy xf86-video-ati xf86-video-amdgpu xf86-video-intel xf86-video-nouveau vulkan-intel vulkan-radeon libva-mesa-driver libva-vdpau-driver mesa-vdpau lib32-vulkan-intel lib32-vulkan-radeon lib32-libva-vdpau-driver lib32-mesa-vdpau’
[2021-06-14T20:10:08+0000] [PACMAN] synchronizing package lists
[2021-06-14T20:10:08+0000] [ALPM] transaction started
[2021-06-14T20:10:08+0000] [ALPM] installed xf86-video-ati (1:19.1.0-2)
[2021-06-14T20:10:08+0000] [ALPM] installed xf86-video-amdgpu (19.1.0-2)
[2021-06-14T20:10:08+0000] [ALPM] installed libxvmc (1.0.12-3)
[2021-06-14T20:10:08+0000] [ALPM] installed xf86-video-intel (1:2.99.917+908+g7181c5a4-1)
[2021-06-14T20:10:08+0000] [ALPM-SCRIPTLET] >>> This driver now uses DRI3 as the default Direct Rendering
[2021-06-14T20:10:08+0000] [ALPM-SCRIPTLET] Infrastructure. You can try falling back to DRI2 if you run
[2021-06-14T20:10:08+0000] [ALPM-SCRIPTLET] into trouble. To do so, save a file with the following
[2021-06-14T20:10:08+0000] [ALPM-SCRIPTLET] content as /etc/X11/xorg.conf.d/20-intel.conf :
[2021-06-14T20:10:08+0000] [ALPM-SCRIPTLET] Section “Device”
[2021-06-14T20:10:08+0000] [ALPM-SCRIPTLET] Identifier “Intel Graphics”
[2021-06-14T20:10:08+0000] [ALPM-SCRIPTLET] Driver “intel”
[2021-06-14T20:10:08+0000] [ALPM-SCRIPTLET] Option “DRI” “2” # DRI3 is now default
[2021-06-14T20:10:08+0000] [ALPM-SCRIPTLET] #Option “AccelMethod” “sna” # default
[2021-06-14T20:10:08+0000] [ALPM-SCRIPTLET] #Option “AccelMethod” “uxa” # fallback
[2021-06-14T20:10:08+0000] [ALPM-SCRIPTLET] EndSection
[2021-06-14T20:10:08+0000] [ALPM] installed xf86-video-nouveau (1.0.16-2)
[2021-06-14T20:10:08+0000] [ALPM] installed vulkan-intel (20.1.8-1)
[2021-06-14T20:10:08+0000] [ALPM] installed vulkan-radeon (20.1.8-1)
[2021-06-14T20:10:08+0000] [ALPM] installed mesa-vdpau (20.1.8-1)
[2021-06-14T20:10:08+0000] [ALPM] installed lib32-vulkan-intel (20.1.8-1)
[2021-06-14T20:10:08+0000] [ALPM] installed lib32-vulkan-radeon (20.1.8-1)
[2021-06-14T20:10:08+0000] [ALPM] installed lib32-mesa-vdpau (20.1.8-1)
[2021-06-14T20:10:11+0000] [ALPM] transaction completed
[2021-06-14T20:10:11+0000] [ALPM] running ‘30-systemd-update.hook’…
[2021-06-14T20:10:11+0000] [PACMAN] Running ‘pacman --noconfirm --cachedir /var/cache/pacman/pkg --config /opt/mhwd/pacman-mhwd.conf --root / -D --asexplicit xf86-video-ati xf86-video-amdgpu xf86-video-intel xf86-video-nouveau vulkan-intel vulkan-radeon libva-mesa-driver libva-vdpau-driver mesa-vdpau lib32-vulkan-intel lib32-vulkan-radeon lib32-libva-vdpau-driver lib32-mesa-vdpau’
[2021-06-14T20:46:27+0000] [PACMAN] Running ‘pacman -Syu’
[2021-06-14T20:46:27+0000] [PACMAN] synchronizing package lists
[2021-06-14T20:49:18+0000] [PACMAN] Running ‘pacman -Syu pacman-mirrorlist’
[2021-06-14T20:49:18+0000] [PACMAN] synchronizing package lists
[2021-06-14T20:54:16+0000] [PACMAN] Running ‘pacman -Syu’
[2021-06-14T20:54:16+0000] [PACMAN] synchronizing package lists
[2021-06-14T20:54:44+0000] [PACMAN] starting full system upgrade

Your update worked fine. Only the kernel is not working with the systemd update. Either try to use linux414 or any other kernel. We have to see if we can fix linux419 or not.

I really appreciate your help. Can I use what you posted earlier to get a new kernel? sudo mhwd-kernel -i linux510

Yes. Replace linux510 with any other kernel you like. mhwd-kernel -l will list you the available options.

I filed an issue for upstream:

Sorry for my ignorance…it didn’t work for me. I should be able to do this while booting from the USB right? Is there some extra step I need to do?

You have to chroot into your installed system. Also your pacman.log was from the USB stick, not your system.

I just tested some more kernels. 4.4 booted fine on my end. 4.9 not. So it might not only be the 4.19 series affected by systemd 248

I see. OK I will give this a try again.

Seems this is related somehow:

CHANGES WITH 247:

    * KERNEL API INCOMPATIBILITY: Linux 4.14 introduced two new uevents
      "bind" and "unbind" to the Linux device model. When this kernel
      change was made, systemd-udevd was only minimally updated to handle
      and propagate these new event types. The introduction of these new
      uevents (which are typically generated for USB devices and devices
      needing a firmware upload before being functional) resulted in a
      number of issues which we so far didn't address. We hoped the kernel
      maintainers would themselves address these issues in some form, but
      that did not happen. To handle them properly, many (if not most) udev
      rules files shipped in various packages need updating, and so do many
      programs that monitor or enumerate devices with libudev or sd-device,
      or otherwise process uevents. Please note that this incompatibility
      is not fault of systemd or udev, but caused by an incompatible kernel
      change that happened back in Linux 4.14, but is becoming more and
      more visible as the new uevents are generated by more kernel drivers.

      To minimize issues resulting from this kernel change (but not avoid
      them entirely) starting with systemd-udevd 247 the udev "tags"
      concept (which is a concept for marking and filtering devices during
      enumeration and monitoring) has been reworked: udev tags are now
      "sticky", meaning that once a tag is assigned to a device it will not
      be removed from the device again until the device itself is removed
      (i.e. unplugged). This makes sure that any application monitoring
      devices that match a specific tag is guaranteed to both see uevents
      where the device starts being relevant, and those where it stops
      being relevant (the latter now regularly happening due to the new
      "unbind" uevent type). The udev tags concept is hence now a concept
      tied to a *device* instead of a device *event* — unlike for example
      udev properties whose lifecycle (as before) is generally tied to a
      device event, meaning that the previously determined properties are
      forgotten whenever a new uevent is processed.

      With the newly redefined udev tags concept, sometimes it's necessary
      to determine which tags are the ones applied by the most recent
      uevent/database update, in order to discern them from those
      originating from earlier uevents/database updates of the same
      device. To accommodate for this a new automatic property CURRENT_TAGS
      has been added that works similar to the existing TAGS property but
      only lists tags set by the most recent uevent/database
      update. Similarly, the libudev/sd-device API has been updated with
      new functions to enumerate these 'current' tags, in addition to the
      existing APIs that now enumerate the 'sticky' ones.

      To properly handle "bind"/"unbind" on Linux 4.14 and newer it is
      essential that all udev rules files and applications are updated to
      handle the new events. Specifically:

      • All rule files that currently use a header guard similar to
        ACTION!="add|change",GOTO="xyz_end" should be updated to use
        ACTION=="remove",GOTO="xyz_end" instead, so that the
        properties/tags they add are also applied whenever "bind" (or
        "unbind") is seen. (This is most important for all physical device
        types — those for which "bind" and "unbind" are currently
        generated, for all other device types this change is still
        recommended but not as important — but certainly prepares for
        future kernel uevent type additions).

      • Similarly, all code monitoring devices that contains an 'if' branch
        discerning the "add" + "change" uevent actions from all other
        uevents actions (i.e. considering devices only relevant after "add"
        or "change", and irrelevant on all other events) should be reworked
        to instead negatively check for "remove" only (i.e. considering
        devices relevant after all event types, except for "remove", which
        invalidates the device). Note that this also means that devices
        should be considered relevant on "unbind", even though conceptually
        this — in some form — invalidates the device. Since the precise
        effect of "unbind" is not generically defined, devices should be
        considered relevant even after "unbind", however I/O errors
        accessing the device should then be handled gracefully.

      • Any code that uses device tags for deciding whether a device is
        relevant or not most likely needs to be updated to use the new
        udev_device_has_current_tag() API (or sd_device_has_current_tag()
        in case sd-device is used), to check whether the tag is set at the
        moment an uevent is seen (as opposed to the existing
        udev_device_has_tag() API which checks if the tag ever existed on
        the device, following the API concept redefinition explained
        above).

      We are very sorry for this breakage and the requirement to update
      packages using these interfaces. We'd again like to underline that
      this is not caused by systemd/udev changes, but result of a kernel
      behaviour change.

    * UPCOMING INCOMPATIBILITY: So far most downstream distribution
      packages have not retriggered devices once the udev package (or any
      auxiliary package installing additional udev rules) is updated. We
      intend to work with major distributions to change this, so that
      "udevadm trigger -a change" is issued on such upgrades, ensuring that
      the updated ruleset is applied to the devices already discovered, so
      that (asynchronously) after the upgrade completed the udev database
      is consistent with the updated rule set. This means udev rules must
      be ready to be retriggered with a "change" action any time, and
      result in correct and complete udev database entries. While the
      majority of udev rule files known to us currently get this right,
      some don't. Specifically, there are udev rules files included in
      various packages that only set udev properties on the "add" action,
      but do not handle the "change" action. If a device matching those
      rules is retriggered with the "change" action (as is intended here)
      it would suddenly lose the relevant properties. This always has been
      problematic, but as soon as all udev devices are triggered on relevant
      package upgrades this will become particularly so. It is strongly
      recommended to fix offending rules so that they can handle a "change"
      action at any time, and acquire all necessary udev properties even
      then. Or in other words: the header guard mentioned above
      (ACTION=="remove",GOTO="xyz_end") is the correct approach to handle
      this, as it makes sure rules are rerun on "change" correctly, and
      accumulate the correct and complete set of udev properties. udev rule
      definitions that cannot handle "change" events being triggered at
      arbitrary times should be considered buggy.
1 Like

I am still having an issue. I think I managed to download a new kernel, but there seem to be issues with doing it in chroot (I get some error message). I am going to have to get back into this tomorrow.

Doing a chroot should be super easy:

[phil@development tmp]$ manjaro-chroot -a
[sudo] password for phil: 
==> Detected systems:
 --> 0) ManjaroLinux
 --> 1) ManjaroLinux1
==> Select system to mount [0-1] : 

Yes I was able to do that. Here is what I get:

[manjaro@manjaro ~]$ sudo manjaro-chroot -a
grub-probe: error: cannot find a GRUB drive for /dev/sdb1. Check your device.map.
grub-probe: error: cannot find a GRUB drive for /dev/sdb1. Check your device.map.
==> Mounting (ManjaroLinux) [/dev/sda2]
→ mount: [/mnt]
→ mount: [/mnt/boot/efi]

Then I ran the mhwd for the kernel and it seem to work, but I got an error message about not having a target

Did you do a sudo pacman -Syy before? You can also try to install it via pamac: pamac install linux510. Else post the exact error you get.

I just did sudo pacman -Syy and then this:

[manjaro /]# sudo mhwd-kernel -i linux510
:: Synchronizing package databases...
 core                                                 169.1 KiB  3.75 MiB/s 00:00 [##############################################] 100%
 extra                                               1905.9 KiB  32.7 MiB/s 00:00 [##############################################] 100%
 community                                              6.6 MiB  50.6 MiB/s 00:00 [##############################################] 100%
 multilib                                             178.2 KiB  58.0 MiB/s 00:00 [##############################################] 100%
error: no targets specified (use -h for help)

Seems to have worked now with the

pamac install linux510

And now my system is back up and running as normal. Thank you so much!

2 Likes