HTML HTML Raw Broken on boot after update... again

5.15 LTS - Encrypted

ERROR: device ‘/dev/mapper/luks-c0949ed2-cf41-4b82-bcb1-4563689838d5’ not found. Skipping fsck.
MOUNT: /new_root: no valid filesystem type specified
ERROR: failed to mount ‘/dev/mapper/luks-c0949ed2-cf41-4b82-bcb1-4563689838d5’ on real root
You are now being dropped into an emergency shell.

Tried to mount:

MOUNT: /new_root: unknown filesystem type ‘crypto_LUKS‘

Tried to chroot, reinstall all packages, reinstall kernel, update-grub, some other methods from this & arch forums with downgrading mkinitcpio, running it with -P after messing with mkinitcpio.conf /etc/default/grub etc.

I’m lost, dunno what else I can do, no more information on this topic. Please help, need working OS ASAP.

Things modified after broken update:

  • decommented GRUB_ENABLE_CRYPTODISK=y
  • added encrypt to HOOKS (base udev autodetect modconf keyboard keymap consolefont block encrypt filesystems fsck)

Compare the contents of /etc/fstab to your actual devices UUIDs

If you dont know … then show us, ex;

sudo blkid
cat /etc/fstab

/etc/fstab

UUID=C95C-89A9  /boot/efi       vfat    umask=0077      0       2
UUID=9072be81-7fab-440f-9f21-40c328c93b2c       /       ext4    defaults,noatime        0       1
/dev/mapper/luks-85a84450-3cc8-464b-aeaf-96b04e3178e0   swap    swap    defaults,noatime        0       0
tmpfs   /tmp    tmpfs   defaults,noatime,mode=1777      0       0
UUID=9072be81-7fab-440f-9f21-40c328c93b2c       /boot   ext4    defaults        0       0

blkid

/dev/loop1: BLOCK_SIZE="262144" TYPE="squashfs"
/dev/sdb2: UUID="c0949ed2-cf41-4b82-bcb1-4563689838d5" TYPE="crypto_LUKS" PARTLABEL="root" PARTUUID="a974d905-062a-f942-97c0-4f32a31cabb2"
/dev/sdb3: UUID="85a84450-3cc8-464b-aeaf-96b04e3178e0" TYPE="crypto_LUKS" PARTUUID="b66180e1-5ec6-cc43-9b16-b0926e0137ef"
/dev/sdb1: LABEL_FATBOOT="NO_LABEL" LABEL="NO_LABEL" UUID="C95C-89A9" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="eac9643c-a029-3640-9d12-9455036d70bf"
/dev/loop2: BLOCK_SIZE="262144" TYPE="squashfs"
/dev/loop0: BLOCK_SIZE="262144" TYPE="squashfs"
/dev/mapper/luks-c0949ed2-cf41-4b82-bcb1-4563689838d5: UUID="9072be81-7fab-440f-9f21-40c328c93b2c" BLOCK_SIZE="4096" TYPE="ext4"
/dev/sdc2: SEC_TYPE="msdos" LABEL_FATBOOT="MISO_EFI" LABEL="MISO_EFI" UUID="C08C-67BD" BLOCK_SIZE="512" TYPE="vfat"
/dev/sdc1: BLOCK_SIZE="2048" UUID="2022-08-16-12-45-56-00" LABEL="MANJARO_KDEM_2137" TYPE="iso9660"
/dev/loop3: BLOCK_SIZE="262144" TYPE="squashfs"

I assume this corresponds to the redacted device in the error message.

For now, comment out the line;

/etc/fstab

UUID=C95C-89A9  /boot/efi       vfat    umask=0077      0       2
UUID=9072be81-7fab-440f-9f21-40c328c93b2c       /       ext4    defaults,noatime        0       1
# /dev/mapper/luks-85a84450-3cc8-464b-aeaf-96b04e3178e0   swap    swap    defaults,noatime        0       0
tmpfs   /tmp    tmpfs   defaults,noatime,mode=1777      0       0
UUID=9072be81-7fab-440f-9f21-40c328c93b2c       /boot   ext4    defaults        0       0

(reboot)

Same errors. Checked, it’s /dev/mapper/luks-c0949ed2-cf41-4b82-bcb1-4563689838d5 that corresponds for both errors

Ah, so actually a different device/path. OK.
That one is not listed in fstab.

What about

cat /etc/crypttab
luks-c0949ed2-cf41-4b82-bcb1-4563689838d5       UUID=c0949ed2-cf41-4b82-bcb1-4563689838d5       none    luks,nofail
luks-85a84450-3cc8-464b-aeaf-96b04e3178e0       UUID=85a84450-3cc8-464b-aeaf-96b04e3178e0       /crypto_keyfile.bin   luks,nofail

There it is.

From here I wont be too much help because I’m not that well versed in encryption.

Some entries look weird to me, like

These being the same doesnt look right to me.

But as I said I’m not familiar wit this setup so we might wait for someone who knows luks.

1 Like

Thanks for trying.

It was default encryption setup in Manjaro Installer.

/dev/sdb1 for boot
/dev/sdb2 for root
/dev/sdb3 for swap

This fails because you can’t mount a LUKS device. LUKS is not a filesystem.

Create a folder called /root_partition for a test. Then you have to decrypt the partition first:

cryptsetup luksOpen /dev/sdb2 root_partition

It will ask you for your pass phrase. Next is to list available options via lsblk.

Mount the decrypted one to get to the content:

mount /dev/mapper/root_partition /root_partition

You should find your content in /root_partition folder.


Based on your blkid you have this for your root partition:

/dev/sdb2: UUID="c0949ed2-cf41-4b82-bcb1-4563689838d5" TYPE="crypto_LUKS" PARTLABEL="root" 
/dev/mapper/luks-c0949ed2-cf41-4b82-bcb1-4563689838d5: UUID="9072be81-7fab-440f-9f21-40c328c93b2c" BLOCK_SIZE="4096" TYPE="ext4"

This is the normal way we use with UUID information. To mount the partition it would be then:

mount /dev/mapper/luks-c0949ed2-cf41-4b82-bcb1-4563689838d5 /

However your errored out before with:

So what should I do to permanently fix my setup?

The issue might be this one: https://unix.stackexchange.com/questions/579363/grub-configuration-is-damaged-error-device-dev-mapper-luks-03xxxxxx-not-found

It is an old and known one with grub. So check your boot entries. An older grub version can’t handle dashes like c0949ed2-cf41-4b82-bcb1-4563689838d5. You can try to change that to c0949ed2cf414b82bcb14563689838d5 and see if that boots.


Go to your boot menu, select the entry and press CTRL+E. check for the UUID line and remove the dashes of it. Press CTRL+X to execute. If that helps and you are able too boot call sudo sed -i -e '/cryptomount -u/ {s/-//g;s/ u/ -u/g}' /boot/grub/grub.cfg

To fix it permanently you have to update grub also on your EFI/MBR and not just the grub ALPM package: GRUB/Restore the GRUB Bootloader - Manjaro.

See also: [pkg-upd] 2.06.r456.g65bc45963-6 (6eca32e6) · Commits · Packages / Core / grub · GitLab, Arch Linux - News: Grub bootloader upgrade and configuration incompatibilities


Here is the full story from our older Troubleshoot Guideline:

2023-03-31

GRUB, LUKS and full disk encryption: 'no such cryptodisk found'

Following grub’s update, new configuration may be needed to be manually applied:

  • Based on the manual, unlike filesystem UUIDs, UUIDs for encrypted devices must be specified without dash separators. So check /boot/grub/grub.cfg for entries like cryptomount -u 3722dfb2-3b32-414b-bd59-4329fa92b6a9 and try to remove the dash separators. You have to redo that whenever you update a kernel or call update-grub on your own. You can read more about it here
  • A message “no such cryptodisk found” may appear on the following reboot, but should not prevent you from continuing by typing any key. This happens if you forgot to remove the dashes in /boot/grub/grub.cfg after update-grub got called.

We may add a alpm-hook to grub package to do that monkey job for you automatically. As an alternative you may reinstall grub, as the newer version can handle both cases, including the dashes.

For more details:

So, I’m struggling with what exactly am I trying to change in grub? It was up to date, seems like. GRUB version is 2.12rc1.r49

cryptomount -u c0949ed2cf414b82bcb14563689838d5
                set root='cryptouuid/c0949ed2cf414b82bcb14563689838d5'
                if [ x$feature_platform_search_hint = xy ]; then
                  search --no-floppy --fs-uuid --set=root --hint='cryptouuid/c0949ed2cf414b82bcb14563689838d5'  9072be81-7fab-440f-9f21-40c328c93b2c
                else
                  search --no-floppy --fs-uuid --set=root 9072be81-7fab-440f-9f21-40c328c93b2c
                fi
                linux   /boot/vmlinuz-5.15-x86_64 root=UUID=9072be81-7fab-440f-9f21-40c328c93b2c rw  quiet nvme_load=YES cryptdevice=UUID=c0949ed2-cf41-4b82-bcb1-4563689838d5:luks-c0949ed2-cf41-4b82-bcb1-4563689838d5:allow-discards rd.luks.options=allow-discards root=/dev/mapper/luks-c0949ed2-cf41-4b82-bcb1-4563689838d5 apparmor=1 security=apparmor udev.log_priority=3

Your grub entry looks fine as it has no dashes. However something changed:

/etc/fstab

UUID=9072be81-7fab-440f-9f21-40c328c93b2c       /       ext4    defaults,noatime        0       1
/dev/mapper/luks-85a84450-3cc8-464b-aeaf-96b04e3178e0   swap    swap    defaults,noatime        0       0

blkid

/dev/sdb2: UUID="c0949ed2-cf41-4b82-bcb1-4563689838d5" TYPE="crypto_LUKS" PARTLABEL="root" 
/dev/mapper/luks-c0949ed2-cf41-4b82-bcb1-4563689838d5: UUID="9072be81-7fab-440f-9f21-40c328c93b2c" BLOCK_SIZE="4096" TYPE="ext4"
/dev/sdb3: UUID="85a84450-3cc8-464b-aeaf-96b04e3178e0" TYPE="crypto_LUKS" PARTUUID="b66180e1-5ec6-cc43-9b16-b0926e0137ef"

Fstab matches you swap partition, however not your root. So check which UUID for your root partition is correct and fix that in grub.cfg.


UUID=9072be81-7fab-440f-9f21-40c328c93b2c       /       ext4    defaults,noatime        0       1

… seems wrong and this might help:

/dev/mapper/luks-c0949ed2-cf41-4b82-bcb1-4563689838d5       /       ext4    defaults,noatime        0       1

Else check if the other UUID is correct and switch it around. Have to check how fstab entries for luks normally look like. Also remember if and why you have changed system configuration files.

So In previous message was the grub.cfg & everything seems like matching, right? I’m not sure what I should do to help you to figure it out.

If I compare the UUIDs of fstab and the entries of grub and blkid, that dont match, so it needs fixing.

Ran update-grub > reboot.
Error messages have changed to

ERROR: device ‘UUID=9072be81-7fab-440f-9f21-40c328c93b2c’ not found. Skipping fsck.
MOUNT: /new_root: can't find UUID=9072be81-7fab-440f-9f21-40c328c93b2c.
ERROR: failed to mount ‘UUID=9072be81-7fab-440f-9f21-40c328c93b2c’ on real root
You are now being dropped into an emergency shell.
sh: can't access tty: job control turned off
[rootfs ~]#

fstab still same

UUID=C95C-89A9  /boot/efi       vfat    umask=0077      0       2
UUID=9072be81-7fab-440f-9f21-40c328c93b2c       /       ext4    defaults,noatime        0       1
/dev/mapper/luks-85a84450-3cc8-464b-aeaf-96b04e3178e0   swap    swap    defaults,noatime        0       0
tmpfs   /tmp    tmpfs   defaults,noatime,mode=1777      0       0
UUID=9072be81-7fab-440f-9f21-40c328c93b2c       /boot   ext4    defaults        0       0

blkid

/dev/sdb2: UUID="c0949ed2-cf41-4b82-bcb1-4563689838d5" TYPE="crypto_LUKS" PARTLABEL="root" PARTUUID="a974d905-
062a-f942-97c0-4f32a31cabb2"
/dev/sdb3: UUID="85a84450-3cc8-464b-aeaf-96b04e3178e0" TYPE="crypto_LUKS" PARTUUID="b66180e1-5ec6-cc43-9b16-b0
926e0137ef"
/dev/sdb1: LABEL_FATBOOT="NO_LABEL" LABEL="NO_LABEL" UUID="C95C-89A9" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="e
ac9643c-a029-3640-9d12-9455036d70bf"

/dev/mapper/luks-c0949ed2-cf41-4b82-bcb1-4563689838d5: UUID="9072be81-7fab-440f-9f21-40c328c93b2c" BLOCK_SIZE="4096" TYPE="ext4"

grub entry I’ve booted with

menuentry 'Manjaro Linux (Kernel: 5.15.137-1-MANJARO x64)' --class manjaro --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-5.15.137-1-MANJARO x64-advanced-9072be81-7fab-440f-9f21-40c328c93b2c' {
        savedefault
                load_video
                set gfxpayload=keep
                insmod gzio
                insmod part_gpt
                insmod cryptodisk
                insmod luks
                insmod gcry_rijndael
                insmod gcry_rijndael
                insmod gcry_sha256
                insmod ext2
                cryptomount -u c0949ed2cf414b82bcb14563689838d5
                set root='cryptouuid/c0949ed2cf414b82bcb14563689838d5'
                if [ x$feature_platform_search_hint = xy ]; then
                  search --no-floppy --fs-uuid --set=root --hint='cryptouuid/c0949ed2cf414b82bcb14563689838d5'  9072be81-7fab-440f-9f21-40c328c93b2c
                else
                  search --no-floppy --fs-uuid --set=root 9072be81-7fab-440f-9f21-40c328c93b2c
                fi
                linux   /vmlinuz-5.15-x86_64 root=UUID=9072be81-7fab-440f-9f21-40c328c93b2c rw  quiet udev.log_priority=3
                initrd  /intel-ucode.img /initramfs-5.15-x86_64.img
        }

At this point I don’t understand what I need to do, not being software engineer.

Everything matches now with the /root UUID (except it’s LUKS which can’t be mounted).

Again, your fstab is not correct. Try either

/dev/mapper/luks-c0949ed2-cf41-4b82-bcb1-4563689838d5       /       ext4    defaults,noatime        0       1

or

UUID=c0949ed2-cf41-4b82-bcb1-4563689838d5       /       ext4    defaults,noatime        0       1

And remember how that got changed and why …

blkid states: /dev/sdb2: UUID="c0949ed2-cf41-4b82-bcb1-4563689838d5" TYPE="crypto_LUKS" PARTLABEL="root" PARTUUID="a974d905- 062a-f942-97c0-4f32a31cabb2" for your root partition and not UUID=9072be81-7fab-440f-9f21-40c328c93b2


Grub tells the system where the kernel and root partition is located, then it starts a bare minimum Linux system. That reads fstab and actually mounts the partition. If in the chain grub.cfg > fstab some mismatch exists it will bail out. Errors can also happen if /etc/default/grub has a false entry, as that is the base to generate your grub.cfg file.

Changing didn’t work. In grub linux root= to both ‘/dev/mapper/luks-c0949ed2-cf41-4b82-bcb1-4563689838d5’ and ‘UUID=c0949ed2-cf41-4b82-bcb1-4563689838d5’.