Can't boot after latest update- Timed out waiting for device /dev/mapper/vg0-rootlv

After the latest update (stable branch) I can’t boot, and am at a loss for how to fix it. The update was rather large at 3GB, and lsb-release says 24.0.0, if that’s correct. Here’s what I see at the console on boot (abbreviated, as I’m typing all this in from a photo):

...
Please enter passphrase for disk SAMSUNG ... (crypt0): .....
[ OK ] Finished Cryptography Setup for crypt0.
[ OK ] Reached target Local Encrypted Volumes.
[ OK ] Reached target System Initialization.
[ OK ] Reached target Basic System.
*(here a timeout occurs after 90 sec or so)*
[ TIME ] Timed out waiting for device /dev/mapper/vg0-rootlv.
[ DEPEND ] Dependency failed for /sysroot.
[ DEPEND ] Dependency failed for Initrd Root File System.
[ DEPEND ] Dependency failed for Mountpoints Configured in the Real Root.
[ DEPEND ] Dependency failed for Initrd Root Device.
[ DEPEND ] Dependency failed for File System Check on /dev/mapper/vg0-rootlv.
[ OK ] Stopped target Basic System.
[ OK ] Reached target Initrd File Systems.
[ OK ] Stopped target System Initialization.
[ OK ] Stopping Dispatch Password Requests to Console...
[ OK ] Stopped Dispatch Password Requests to Console.
[ OK ] Stopped Dispatch Password Requests to Console Directory Watch.
[ OK ] Started Emergency Shell.
[ OK ] Reached target Emergency Mode.
You are in emergency mode. After logging in, type "journalctl -xb" to view
system logs, "systemctl reboot" to reboot, or "exit"
to continue bootup.

Cannot open access to console, the root account is locked.
See sulogin(8) man page for more details.

Press Enter to continue

Pressing Enter times out after a while and shows the emergency mode message again, so that doesn’t help.

If I recall, there was an error message during the update about not being able to download some packages (didn’t write it down), but I clicked OK to that and the install completed. So it’s possible that I now have partially updated packages.

I am able to boot the live ISO (X86_64 Gnome image), and can unlock my filesystems with cryptsetup and mount them by hand. Obviously, I have an encrypted rootfs. The data appears to be intact and the hardware seems fine.

I then tried e2fsck -c on my root partition. No errors, and no change afterwards.

I then tried the first three options in the System Recovery program, and it found my root and efi partitions, and all options said they completed successfully, but none of them seemed to work. All completed suspiciously fast, and I do not see that the .img files in /mnt/boot/efi were even updated, so I’m not convinced that it did what it said. I then tried System Recovery’s interactive options, like open the package manager, or gather logs, and they all say success, but seem to do nothing.

After a day working on this, I’m stuck, and am at a loss for where to go from here. I have all my data, just can’t boot. Any ideas short of re-installing?

Did you also try to go one step further and chroot
manjaro-chroot /mountpoint /bin/bash
and perhaps look at the logs or re-run the update?

I did check the logs from journalctl -rD /mnt/var/log/journal, but didn’t see anything suspicious there. pacman.log had a lot of messages making it hard to look for relevant errors.

Anyway, good suggestion to try to re-run the upgrade from a chroot. At first, I did it and got this:

# pacman -Syu
:: Synchronizing package databases...
 core is up to date
 extra                                                          8.3 MiB  4.75 MiB/s 00:02 [####################################################] 100%
 community is up to date
 multilib is up to date
error: failed retrieving file 'core.db' from mirror.netzspielplatz.de : The requested URL returned error: 502
error: failed retrieving file 'extra.db' from mirror.netzspielplatz.de : The requested URL returned error: 502
error: failed retrieving file 'community.db' from mirror.netzspielplatz.de : The requested URL returned error: 502
warning: too many errors from mirror.netzspielplatz.de, skipping for the remainder of this transaction
:: Starting full system upgrade...
 there is nothing to do

Obviously the mirror list was bad for some reason, so I fixed that:

# pacman-mirrors
Pacman-mirrors version 4.24.0
Local mirror status for stable branch
 Mirror # 1 https://mirror.netzspielplatz.de/manjaro/packages/ does not exist
 Mirror # 2 https://mirror.23media.com/manjaro/ does not exist
 Mirror # 3 https://manjaro.moson.org/ does not exist
# pacman-mirrors --country Germany
::INFO Downloading mirrors from Manjaro
...
::INFO Querying mirrors - This may take some time
  0.259 Germany        : https://mirror.alpix.eu/manjaro/
  2.674 Germany        : https://ftp.gwdg.de/pub/linux/manjaro/
  0.287 Germany        : https://manjaro.kurdy.org/
::INFO Writing mirror list
::Germany         : https://mirror.alpix.eu/manjaro/stable/$repo/$arch
::Germany         : https://manjaro.kurdy.org/stable/$repo/$arch
::Germany         : https://ftp.gwdg.de/pub/linux/manjaro/stable/$repo/$arch
::INFO Mirror list generated and saved to: /etc/pacman.d/mirrorlist
...

and then re-ran the upgrade:

# pacman -Syu
:: Synchronizing package databases...
 core is up to date
 extra                                                          8.3 MiB  4.37 MiB/s 00:02 [####################################################] 100%
 community is up to date
 multilib is up to date
:: Starting full system upgrade...
 there is nothing to do

So, the upgrade didn’t fix it.

Speaking of logs, two warnings in pacman.log from this morning did catch my eye:

# grep warning /var/log/pacman.log
...
[2024-05-14T08:35:27+0200] [ALPM] warning: /etc/mkinitcpio.conf installed as /etc/mkinitcpio.conf.pacnew
[2024-05-14T08:38:39+0200] [ALPM-SCRIPTLET] /usr/bin/grub-probe: warning: unknown device type nvme0n1.

I wonder if the EFI images got generated incorrectly? They key difference in mkinitcpio.conf is a very different setting for HOOKS, in this abbreviated diff:

# diff /etc/mkinitcpio.conf /etc/mkinitcpio.conf.pacnew
< #    usr, fsck and shutdown hooks.
< HOOKS=(systemd keyboard keymap sd-vconsole block sd-encrypt autodetect modconf sd-lvm2 filesystems fsck)
---
> #    usr and fsck hooks.
> HOOKS=(base udev autodetect microcode modconf kms keyboard keymap consolefont block filesystems fsck)

It’s a long time ago that I used lvm and also that I experimented with converting the HOOKS
in /etc/mkinitcpio.conf to systemd hooks (sd-encrypt vs encrypt for instance)

A thing I noticed when I checked the man page for mkinitcpio.conf:
it stated that
mkinitcpio -L
will list the available HOOKS

sd-lvm2

which you appear to use
is not in that list. :man_shrugging:

Perhaps it used to work like that - and something changed.
I would try to use the lvm2 HOOK instead.

mkinitcpio -P
afterwards
and update-grub for good measure

Yes, this has done it! We were on the same track, as I also noticed that sd-lvm2 didn’t exist when I tried to run mkinitcpio -P.

So, the fundamental problem was that the lvm2 hook previously had a symlink named sd-lvm2, but this was dropped in mkinitcpio v38 (see mkinitcpio changelog and search for sd-lvm2).

While fixing that, I also re-worked my HOOKS to be ordered more like the example in the newly distributed mkinitcpio.conf. Here’s what I ended up with:

HOOKS=(systemd autodetect modconf keyboard keymap sd-vconsole sd-encrypt block lvm2 filesystems fsck)

On top of this, I was having problems with my login screen only showing up about 20% of the time, but that turned out to have nothing to do with mkinitcpio. It was that I had disabled secure boot in my HP ProBook’s firmware to troubleshoot this booting problem, but inadvertently enabled legacy boot mode. That caused the video device to start in some legacy video mode with large text, and it also apparently made amdgpu’s initialization process flaky. Disabling legacy mode in the firmware fixed that.

update-grub didn’t turn out to be necessary, however I’ll still do that and call it a day.

Thanks!

1 Like

It also looks like you haven’t been maintaining your system, as the Community repo was merged into Extra about a year ago. You probably have .pacnew & .pacsave files that need to be looked at and appropriate action taken (merged, deleted, replaced etc). Failure to deal with these files may eventually result in a broken installation.

See System Maintenance in the Manjaro Wiki, and look for the heading Pacnew and Pacsave files to learn more about .pacnew files mentioned.

To find specific information on instances that required manual intervention, look through the previous Stable Announcements.

Great points. I’ve updated my repos, dealt with all the pacsave/pacnew files, and moved config files into .d directories where possible, which also helps to avoid migration problems. This was a case of “not enough time”, but I’ll have to make time for these important tasks…

1 Like

This topic was automatically closed 36 hours after the last reply. New replies are no longer allowed.