Your update worked fine. Only the kernel is not working with the systemd update. Either try to use linux414 or any other kernel. We have to see if we can fix linux419 or not.
I really appreciate your help. Can I use what you posted earlier to get a new kernel? sudo mhwd-kernel -i linux510
Yes. Replace linux510 with any other kernel you like. mhwd-kernel -l
will list you the available options.
I filed an issue for upstream:
Sorry for my ignorance…it didn’t work for me. I should be able to do this while booting from the USB right? Is there some extra step I need to do?
You have to chroot into your installed system. Also your pacman.log was from the USB stick, not your system.
I just tested some more kernels. 4.4 booted fine on my end. 4.9 not. So it might not only be the 4.19 series affected by systemd 248 …
I see. OK I will give this a try again.
Seems this is related somehow:
CHANGES WITH 247:
* KERNEL API INCOMPATIBILITY: Linux 4.14 introduced two new uevents
"bind" and "unbind" to the Linux device model. When this kernel
change was made, systemd-udevd was only minimally updated to handle
and propagate these new event types. The introduction of these new
uevents (which are typically generated for USB devices and devices
needing a firmware upload before being functional) resulted in a
number of issues which we so far didn't address. We hoped the kernel
maintainers would themselves address these issues in some form, but
that did not happen. To handle them properly, many (if not most) udev
rules files shipped in various packages need updating, and so do many
programs that monitor or enumerate devices with libudev or sd-device,
or otherwise process uevents. Please note that this incompatibility
is not fault of systemd or udev, but caused by an incompatible kernel
change that happened back in Linux 4.14, but is becoming more and
more visible as the new uevents are generated by more kernel drivers.
To minimize issues resulting from this kernel change (but not avoid
them entirely) starting with systemd-udevd 247 the udev "tags"
concept (which is a concept for marking and filtering devices during
enumeration and monitoring) has been reworked: udev tags are now
"sticky", meaning that once a tag is assigned to a device it will not
be removed from the device again until the device itself is removed
(i.e. unplugged). This makes sure that any application monitoring
devices that match a specific tag is guaranteed to both see uevents
where the device starts being relevant, and those where it stops
being relevant (the latter now regularly happening due to the new
"unbind" uevent type). The udev tags concept is hence now a concept
tied to a *device* instead of a device *event* — unlike for example
udev properties whose lifecycle (as before) is generally tied to a
device event, meaning that the previously determined properties are
forgotten whenever a new uevent is processed.
With the newly redefined udev tags concept, sometimes it's necessary
to determine which tags are the ones applied by the most recent
uevent/database update, in order to discern them from those
originating from earlier uevents/database updates of the same
device. To accommodate for this a new automatic property CURRENT_TAGS
has been added that works similar to the existing TAGS property but
only lists tags set by the most recent uevent/database
update. Similarly, the libudev/sd-device API has been updated with
new functions to enumerate these 'current' tags, in addition to the
existing APIs that now enumerate the 'sticky' ones.
To properly handle "bind"/"unbind" on Linux 4.14 and newer it is
essential that all udev rules files and applications are updated to
handle the new events. Specifically:
• All rule files that currently use a header guard similar to
ACTION!="add|change",GOTO="xyz_end" should be updated to use
ACTION=="remove",GOTO="xyz_end" instead, so that the
properties/tags they add are also applied whenever "bind" (or
"unbind") is seen. (This is most important for all physical device
types — those for which "bind" and "unbind" are currently
generated, for all other device types this change is still
recommended but not as important — but certainly prepares for
future kernel uevent type additions).
• Similarly, all code monitoring devices that contains an 'if' branch
discerning the "add" + "change" uevent actions from all other
uevents actions (i.e. considering devices only relevant after "add"
or "change", and irrelevant on all other events) should be reworked
to instead negatively check for "remove" only (i.e. considering
devices relevant after all event types, except for "remove", which
invalidates the device). Note that this also means that devices
should be considered relevant on "unbind", even though conceptually
this — in some form — invalidates the device. Since the precise
effect of "unbind" is not generically defined, devices should be
considered relevant even after "unbind", however I/O errors
accessing the device should then be handled gracefully.
• Any code that uses device tags for deciding whether a device is
relevant or not most likely needs to be updated to use the new
udev_device_has_current_tag() API (or sd_device_has_current_tag()
in case sd-device is used), to check whether the tag is set at the
moment an uevent is seen (as opposed to the existing
udev_device_has_tag() API which checks if the tag ever existed on
the device, following the API concept redefinition explained
above).
We are very sorry for this breakage and the requirement to update
packages using these interfaces. We'd again like to underline that
this is not caused by systemd/udev changes, but result of a kernel
behaviour change.
* UPCOMING INCOMPATIBILITY: So far most downstream distribution
packages have not retriggered devices once the udev package (or any
auxiliary package installing additional udev rules) is updated. We
intend to work with major distributions to change this, so that
"udevadm trigger -a change" is issued on such upgrades, ensuring that
the updated ruleset is applied to the devices already discovered, so
that (asynchronously) after the upgrade completed the udev database
is consistent with the updated rule set. This means udev rules must
be ready to be retriggered with a "change" action any time, and
result in correct and complete udev database entries. While the
majority of udev rule files known to us currently get this right,
some don't. Specifically, there are udev rules files included in
various packages that only set udev properties on the "add" action,
but do not handle the "change" action. If a device matching those
rules is retriggered with the "change" action (as is intended here)
it would suddenly lose the relevant properties. This always has been
problematic, but as soon as all udev devices are triggered on relevant
package upgrades this will become particularly so. It is strongly
recommended to fix offending rules so that they can handle a "change"
action at any time, and acquire all necessary udev properties even
then. Or in other words: the header guard mentioned above
(ACTION=="remove",GOTO="xyz_end") is the correct approach to handle
this, as it makes sure rules are rerun on "change" correctly, and
accumulate the correct and complete set of udev properties. udev rule
definitions that cannot handle "change" events being triggered at
arbitrary times should be considered buggy.
I am still having an issue. I think I managed to download a new kernel, but there seem to be issues with doing it in chroot (I get some error message). I am going to have to get back into this tomorrow.
Doing a chroot should be super easy:
[phil@development tmp]$ manjaro-chroot -a
[sudo] password for phil:
==> Detected systems:
--> 0) ManjaroLinux
--> 1) ManjaroLinux1
==> Select system to mount [0-1] :
Yes I was able to do that. Here is what I get:
[manjaro@manjaro ~]$ sudo manjaro-chroot -a
grub-probe: error: cannot find a GRUB drive for /dev/sdb1. Check your device.map.
grub-probe: error: cannot find a GRUB drive for /dev/sdb1. Check your device.map.
==> Mounting (ManjaroLinux) [/dev/sda2]
→ mount: [/mnt]
→ mount: [/mnt/boot/efi]
Then I ran the mhwd for the kernel and it seem to work, but I got an error message about not having a target
Did you do a sudo pacman -Syy
before? You can also try to install it via pamac: pamac install linux510
. Else post the exact error you get.
I just did sudo pacman -Syy and then this:
[manjaro /]# sudo mhwd-kernel -i linux510
:: Synchronizing package databases...
core 169.1 KiB 3.75 MiB/s 00:00 [##############################################] 100%
extra 1905.9 KiB 32.7 MiB/s 00:00 [##############################################] 100%
community 6.6 MiB 50.6 MiB/s 00:00 [##############################################] 100%
multilib 178.2 KiB 58.0 MiB/s 00:00 [##############################################] 100%
error: no targets specified (use -h for help)
Seems to have worked now with the
pamac install linux510
And now my system is back up and running as normal. Thank you so much!
I deplore to say I have the same issue with the 4.19 kernel and latest stable update. Running the 4.14 kernel works.
Edit: And yay! this is the first 5.10 series that allows me to see the Plasma desktop … (for unknown reasons Plasma did not start with any Manjaro series 5 kernel, so that combination of kernel 5.10 and Plasma 5.21.5 does the trick on this machine).
Seems 4.9 I can start in VirtualBox. So only 4.19 kernel series won’t boot with systemd v248. I also tested latest git version of systemd. Won’t boot either …
Any idea how to fix this without a live USB? I’m currently at a data center, thought I had my live USB in my backup, updated packages, and broke my laptop.
I can get to the command line, but the only network interface I seem to have is loopback. I wasn’t sure if I could bring up the enp0xxxxx interface that normally comes up, install 510 kernel, update packages, and reboot into the 510 kernel.
I was able to fix it, for some reason my ethernet kernel module wasn’t loading so I did the following:
- I was eventually dropped into a root shell, but had limited services and the only network interface was loopback
- I had access to ethernet for my laptop so I plugged it in, still only loopback
- Checked ethernet controller with
lspci -v | less
Ethernet controller: Intel Corporation Ethernet Connection I219-LM (rev 21)
Subsystem: Hewlett-Packard Company EliteBook 840 G3
Flags: bus master, fast devsel, latency 0, IRQ 132
Memory at e1200000 (32-bit, non-prefetchable) [size=128K]
Capabilities: <access denied>
Kernel driver in use: e1000e
Kernel modules: e1000e
- I then checked to see if the module was loaded:
dmesg | grep e1000e
it was not. Add ethernet module to the kernel:
modprobe e1000e
- Next is setting up networking, which was a little different for me since I was at a datacenter, but basically using
ip
commands you’d do something like the following:
ip link set dev eth0 up
ip addr add 192.168.1.6/24 dev eth0
ip route add default via 192.168.1.1 dev eth0
Likely if you’re at home your gateway is your nameserver and if you’re at a datacenter or somewhere else, you’ll know what those are and their addresses - Last was following up above to install a newer kernel:
mhwd-kernel -i linux510
then double check packages with pacman:
pacman -Syy
and finally reboot:
reboot
Hopefully this helps anyone else who gets stuck without a live USB
Hello,
I also run into this issue this morning. I have to use 4.19 and cannot use newer kernels as they randomly freeze/crash my system because of my Intel CPU. So I was happy that I could use 4.19 now for a long time.
The downgrade to 4.14 fixed the current issue. But I dont like the idea to have to switch even more older kernel now
Wait for the fix coming probably sooner than later.