The [Stable Update] 2021-06-14 prevent me from booting my system

Your update worked fine. Only the kernel is not working with the systemd update. Either try to use linux414 or any other kernel. We have to see if we can fix linux419 or not.

I really appreciate your help. Can I use what you posted earlier to get a new kernel? sudo mhwd-kernel -i linux510

Yes. Replace linux510 with any other kernel you like. mhwd-kernel -l will list you the available options.

I filed an issue for upstream:

Sorry for my ignorance…it didn’t work for me. I should be able to do this while booting from the USB right? Is there some extra step I need to do?

You have to chroot into your installed system. Also your pacman.log was from the USB stick, not your system.

I just tested some more kernels. 4.4 booted fine on my end. 4.9 not. So it might not only be the 4.19 series affected by systemd 248

I see. OK I will give this a try again.

Seems this is related somehow:

CHANGES WITH 247:

    * KERNEL API INCOMPATIBILITY: Linux 4.14 introduced two new uevents
      "bind" and "unbind" to the Linux device model. When this kernel
      change was made, systemd-udevd was only minimally updated to handle
      and propagate these new event types. The introduction of these new
      uevents (which are typically generated for USB devices and devices
      needing a firmware upload before being functional) resulted in a
      number of issues which we so far didn't address. We hoped the kernel
      maintainers would themselves address these issues in some form, but
      that did not happen. To handle them properly, many (if not most) udev
      rules files shipped in various packages need updating, and so do many
      programs that monitor or enumerate devices with libudev or sd-device,
      or otherwise process uevents. Please note that this incompatibility
      is not fault of systemd or udev, but caused by an incompatible kernel
      change that happened back in Linux 4.14, but is becoming more and
      more visible as the new uevents are generated by more kernel drivers.

      To minimize issues resulting from this kernel change (but not avoid
      them entirely) starting with systemd-udevd 247 the udev "tags"
      concept (which is a concept for marking and filtering devices during
      enumeration and monitoring) has been reworked: udev tags are now
      "sticky", meaning that once a tag is assigned to a device it will not
      be removed from the device again until the device itself is removed
      (i.e. unplugged). This makes sure that any application monitoring
      devices that match a specific tag is guaranteed to both see uevents
      where the device starts being relevant, and those where it stops
      being relevant (the latter now regularly happening due to the new
      "unbind" uevent type). The udev tags concept is hence now a concept
      tied to a *device* instead of a device *event* — unlike for example
      udev properties whose lifecycle (as before) is generally tied to a
      device event, meaning that the previously determined properties are
      forgotten whenever a new uevent is processed.

      With the newly redefined udev tags concept, sometimes it's necessary
      to determine which tags are the ones applied by the most recent
      uevent/database update, in order to discern them from those
      originating from earlier uevents/database updates of the same
      device. To accommodate for this a new automatic property CURRENT_TAGS
      has been added that works similar to the existing TAGS property but
      only lists tags set by the most recent uevent/database
      update. Similarly, the libudev/sd-device API has been updated with
      new functions to enumerate these 'current' tags, in addition to the
      existing APIs that now enumerate the 'sticky' ones.

      To properly handle "bind"/"unbind" on Linux 4.14 and newer it is
      essential that all udev rules files and applications are updated to
      handle the new events. Specifically:

      • All rule files that currently use a header guard similar to
        ACTION!="add|change",GOTO="xyz_end" should be updated to use
        ACTION=="remove",GOTO="xyz_end" instead, so that the
        properties/tags they add are also applied whenever "bind" (or
        "unbind") is seen. (This is most important for all physical device
        types — those for which "bind" and "unbind" are currently
        generated, for all other device types this change is still
        recommended but not as important — but certainly prepares for
        future kernel uevent type additions).

      • Similarly, all code monitoring devices that contains an 'if' branch
        discerning the "add" + "change" uevent actions from all other
        uevents actions (i.e. considering devices only relevant after "add"
        or "change", and irrelevant on all other events) should be reworked
        to instead negatively check for "remove" only (i.e. considering
        devices relevant after all event types, except for "remove", which
        invalidates the device). Note that this also means that devices
        should be considered relevant on "unbind", even though conceptually
        this — in some form — invalidates the device. Since the precise
        effect of "unbind" is not generically defined, devices should be
        considered relevant even after "unbind", however I/O errors
        accessing the device should then be handled gracefully.

      • Any code that uses device tags for deciding whether a device is
        relevant or not most likely needs to be updated to use the new
        udev_device_has_current_tag() API (or sd_device_has_current_tag()
        in case sd-device is used), to check whether the tag is set at the
        moment an uevent is seen (as opposed to the existing
        udev_device_has_tag() API which checks if the tag ever existed on
        the device, following the API concept redefinition explained
        above).

      We are very sorry for this breakage and the requirement to update
      packages using these interfaces. We'd again like to underline that
      this is not caused by systemd/udev changes, but result of a kernel
      behaviour change.

    * UPCOMING INCOMPATIBILITY: So far most downstream distribution
      packages have not retriggered devices once the udev package (or any
      auxiliary package installing additional udev rules) is updated. We
      intend to work with major distributions to change this, so that
      "udevadm trigger -a change" is issued on such upgrades, ensuring that
      the updated ruleset is applied to the devices already discovered, so
      that (asynchronously) after the upgrade completed the udev database
      is consistent with the updated rule set. This means udev rules must
      be ready to be retriggered with a "change" action any time, and
      result in correct and complete udev database entries. While the
      majority of udev rule files known to us currently get this right,
      some don't. Specifically, there are udev rules files included in
      various packages that only set udev properties on the "add" action,
      but do not handle the "change" action. If a device matching those
      rules is retriggered with the "change" action (as is intended here)
      it would suddenly lose the relevant properties. This always has been
      problematic, but as soon as all udev devices are triggered on relevant
      package upgrades this will become particularly so. It is strongly
      recommended to fix offending rules so that they can handle a "change"
      action at any time, and acquire all necessary udev properties even
      then. Or in other words: the header guard mentioned above
      (ACTION=="remove",GOTO="xyz_end") is the correct approach to handle
      this, as it makes sure rules are rerun on "change" correctly, and
      accumulate the correct and complete set of udev properties. udev rule
      definitions that cannot handle "change" events being triggered at
      arbitrary times should be considered buggy.
1 Like

I am still having an issue. I think I managed to download a new kernel, but there seem to be issues with doing it in chroot (I get some error message). I am going to have to get back into this tomorrow.

Doing a chroot should be super easy:

[phil@development tmp]$ manjaro-chroot -a
[sudo] password for phil: 
==> Detected systems:
 --> 0) ManjaroLinux
 --> 1) ManjaroLinux1
==> Select system to mount [0-1] : 

Yes I was able to do that. Here is what I get:

[manjaro@manjaro ~]$ sudo manjaro-chroot -a
grub-probe: error: cannot find a GRUB drive for /dev/sdb1. Check your device.map.
grub-probe: error: cannot find a GRUB drive for /dev/sdb1. Check your device.map.
==> Mounting (ManjaroLinux) [/dev/sda2]
→ mount: [/mnt]
→ mount: [/mnt/boot/efi]

Then I ran the mhwd for the kernel and it seem to work, but I got an error message about not having a target

Did you do a sudo pacman -Syy before? You can also try to install it via pamac: pamac install linux510. Else post the exact error you get.

I just did sudo pacman -Syy and then this:

[manjaro /]# sudo mhwd-kernel -i linux510
:: Synchronizing package databases...
 core                                                 169.1 KiB  3.75 MiB/s 00:00 [##############################################] 100%
 extra                                               1905.9 KiB  32.7 MiB/s 00:00 [##############################################] 100%
 community                                              6.6 MiB  50.6 MiB/s 00:00 [##############################################] 100%
 multilib                                             178.2 KiB  58.0 MiB/s 00:00 [##############################################] 100%
error: no targets specified (use -h for help)

Seems to have worked now with the

pamac install linux510

And now my system is back up and running as normal. Thank you so much!

2 Likes

I deplore to say I have the same issue with the 4.19 kernel and latest stable update. Running the 4.14 kernel works.

Edit: And yay! this is the first 5.10 series that allows me to see the Plasma desktop … (for unknown reasons Plasma did not start with any Manjaro series 5 kernel, so that combination of kernel 5.10 and Plasma 5.21.5 does the trick on this machine).

1 Like

Seems 4.9 I can start in VirtualBox. So only 4.19 kernel series won’t boot with systemd v248. I also tested latest git version of systemd. Won’t boot either …

2 Likes

Any idea how to fix this without a live USB? I’m currently at a data center, thought I had my live USB in my backup, updated packages, and broke my laptop.

I can get to the command line, but the only network interface I seem to have is loopback. I wasn’t sure if I could bring up the enp0xxxxx interface that normally comes up, install 510 kernel, update packages, and reboot into the 510 kernel.

I was able to fix it, for some reason my ethernet kernel module wasn’t loading so I did the following:

  • I was eventually dropped into a root shell, but had limited services and the only network interface was loopback
  • I had access to ethernet for my laptop so I plugged it in, still only loopback
  • Checked ethernet controller with
lspci -v | less

Ethernet controller: Intel Corporation Ethernet Connection I219-LM (rev 21)
        Subsystem: Hewlett-Packard Company EliteBook 840 G3
        Flags: bus master, fast devsel, latency 0, IRQ 132
        Memory at e1200000 (32-bit, non-prefetchable) [size=128K]
        Capabilities: <access denied>
        Kernel driver in use: e1000e
        Kernel modules: e1000e
  • I then checked to see if the module was loaded:
    dmesg | grep e1000e
    it was not. Add ethernet module to the kernel:
    modprobe e1000e
  • Next is setting up networking, which was a little different for me since I was at a datacenter, but basically using ip commands you’d do something like the following:
    ip link set dev eth0 up
    ip addr add 192.168.1.6/24 dev eth0
    ip route add default via 192.168.1.1 dev eth0
    Likely if you’re at home your gateway is your nameserver and if you’re at a datacenter or somewhere else, you’ll know what those are and their addresses
  • Last was following up above to install a newer kernel:
    mhwd-kernel -i linux510
    then double check packages with pacman:
    pacman -Syy
    and finally reboot:
    reboot

Hopefully this helps anyone else who gets stuck without a live USB

2 Likes

Hello,
I also run into this issue this morning. I have to use 4.19 and cannot use newer kernels as they randomly freeze/crash my system because of my Intel CPU. So I was happy that I could use 4.19 now for a long time.

The downgrade to 4.14 fixed the current issue. But I dont like the idea to have to switch even more older kernel now :frowning:

Wait for the fix coming probably sooner than later.