Why is my Grub + LUKS configuration breaking all the time again?

Hello everyone,
I’m really annoyed that the latest update (I’m on Manjaro unstable) broke my grub configuration again.
My system is using an encrypted (LUKS) root partition.

After I rebooted my system, this screen randomly appeared :

I had this at least 3 times now during my manjaro time, and usually I fixed it by booting from a live USB stick, manually mounting boot /efi partition, chrooting into it and using something like update-grub or install-grub to fix the grub configuration (don’t remember the exact steps anymore that I used the last time).
I usually follow this guide from the manjaro wiki : GRUB/Restore the GRUB Bootloader - Manjaro
Then I can normally reboot the device and it works just fine for a few months usually.

Until for some reason I don’t understand after a while the grub configuration breaks again and something like in the screenshot from above appears again.
This time it happened while I was on the go, so quite difficult to find a USB stick and flash a live OS onto it.
I suspect my configuration to be wrong in some aspect, but I don’t know where exactly.

I made a few Screenshots from the generated grub config (/grub/grub.cfg) which I was able to access from the rescue shell.
Maybe the contain some helpful information already. Otherwise I will follow up with the grub configuration from /etc/default/grub later, when I have access to said live OS.

Any hints what could go wrong here would be greatly appreciated.
Sorry for the bad image quality (on the go as I said)

Because this …

… is not possible to be integrated into the triggered update process as it would be too complicated to automate it for all kind of different configurations. It’s the price for the great flexibilty of customizing the system.

1 Like

There should be nothing to fix - or any reason to break a currently working setup.

Where you able to “fix it”?

Or are you asking for assistance?

My (not your) configuration is … working.

So:
we would likely need to see your’s in order to assess your issue

No I haven’t fixed it yet, because I wasn’t able to get my hands on a USB stick with a live image on it.
If I understand it correctly, there is no way to get the system running again without it, right?

So yes, I am asking for assistance on how I should go ahead this time, to get a configuration that is working persistently.
So basically if the steps in the manjaro system are the way to go.

Which configuration files would you need in order to see, what s going wrong?

that is not remotely difficult.

Once you have:

boot the system or from USB
and provide the output of:
inxi -zv8
at least

information is needed …

There you go: inxi -zv8 - Pastebin.com
I found a USB stick, booted a live image from it and executed the command there.

And since i was able to mount the encrypted partition and chroot into it, here is the content from my /etc/default/grub file:

cat /etc/default/grub
# GRUB boot loader configuration

GRUB_DEFAULT=saved
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="Manjaro"
GRUB_CMDLINE_LINUX_DEFAULT="quiet cryptdevice=UUID=e30b42ae-6a14-4c68-a4ba-6cca0d90f151:luks-e30b42ae-6a14-4c68-a4ba-6cca0d90f151 root=/dev/mapper/luks-e30b42ae-6a14-4c68-a4ba-6cca0d90f151 splash apparmor=1 security=apparmor udev.log_priority=3"
GRUB_CMDLINE_LINUX=""

# Preload both GPT and MBR modules so that they are not missed
GRUB_PRELOAD_MODULES="part_gpt part_msdos"

# Uncomment to enable booting from LUKS encrypted devices
#GRUB_ENABLE_CRYPTODISK=y

# Set to 'countdown' or 'menu' to change timeout behavior,
# press ESC key to display menu.
GRUB_TIMEOUT_STYLE=menu

# Uncomment to use basic console
#GRUB_TERMINAL_INPUT=console

# Uncomment to disable graphical terminal
#GRUB_TERMINAL_OUTPUT=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command 'videoinfo'
GRUB_GFXMODE=3840x2160x31,auto

# Uncomment to allow the kernel use the same resolution used by grub
GRUB_GFXPAYLOAD_LINUX=keep

# Uncomment if you want GRUB to pass to the Linux kernel the old parameter
# format "root=/dev/xxx" instead of "root=/dev/disk/by-uuid/xxx"
#GRUB_DISABLE_LINUX_UUID=true

# Uncomment to disable generation of recovery mode menu entries
GRUB_DISABLE_RECOVERY=true

# Uncomment and set to the desired menu colors.  Used by normal and wallpaper
# modes only.  Entries specified as foreground/background.
GRUB_COLOR_NORMAL="light-gray/black"
GRUB_COLOR_HIGHLIGHT="green/black"

# Uncomment one of them for the gfx desired, a image background or a gfxtheme
#GRUB_BACKGROUND="/usr/share/grub/background.png"
GRUB_THEME="/boot/grub/themes/whitesur-color-4k/theme.txt"

# Uncomment to get a beep at GRUB start
#GRUB_INIT_TUNE="480 440 1"

# Uncomment to make GRUB remember the last selection. This requires
# setting 'GRUB_DEFAULT=saved' above.
GRUB_SAVEDEFAULT=true

# Uncomment to disable submenus in boot menu
#GRUB_DISABLE_SUBMENU=y

# Uncomment this option to enable os-prober execution in the grub-mkconfig command
GRUB_DISABLE_OS_PROBER=false

# Uncomment to ensure that the root filesystem is mounted read-only so that
# systemd-fsck can run the check automatically. We use 'fsck' by default, which
# needs 'rw' as boot parameter, to avoid delay in boot-time. 'fsck' needs to be
# removed from 'mkinitcpio.conf' to make 'systemd-fsck' work.
# See also Arch-Wiki: https://wiki.archlinux.org/index.php/Fsck#Boot_time_checking
#GRUB_ROOT_FS_RO=true

Next I will try to re-install Grub and update the configuration according to the Manjaro Wiki.

Edit: Now that I read through the disabled options: Is it maybe required to enable GRUB_ENABLE_CRYPTODISK?

You will have to know the root cause to be able to answer that question.

One has to ask - what makes your system different?

It depends - if you have a separate unencrypted /boot or EFI partition, then no.

ID-4: /dev/nvme0n1p2 maj-min: 259:2 size: 1024 MiB fs: ext4
label: linux boot uuid: 6542643d-e55f-448e-9edf-7974b392716a


If the whole system is encrypted, including the EFI partition, then yes.
update-grub after any change
or
grub-mkconfig -o /boot/grub/grub.cfg

I’d definitely try it.

Sure, that’s what I’m currently trying to find out :+1:

Edit:

I don’t think my system is that special. I am dual booting Manjaro with Windows and have an encrypted root partition.
That should be it.

Hmm okay, than it’s probably not the root cause, because my /boot and EFI partition are not encrypted.
I would also wondering, how I was able to boot the last years, if this option was required.
I will try the other two commands.

/etc/mkinitcpio.conf

Is the “encrypt” keyword in the HOOKS line?
grep ^HOOKS /etc/mkinitcpio.conf

Actually it isn’t:

grep ^HOOKS /etc/mkinitcpio.conf
HOOKS=(base udev autodetect microcode modconf kms keyboard keymap consolefont block filesystems fsck)

Should I add it?

most definitely

before “filesystems”

mkinitcpio -P
afterwards

grep ^HOOKS /etc/mkinitcpio.conf
HOOKS=(base udev autodetect microcode kms modconf block keyboard keymap consolefont plymouth encrypt filesystems fsck)

The initrd must be able to access the encrypted container - the “encrypt” HOOK put’s the cryptsetup binary in the initrd - without it the container can’t be opened to switch the root over to the main file system.

2 Likes

This part looks strange to me …
Have customized the installation?

please provide the output from

lsblk -f
cat /etc/mkinitcpio.conf

I don’t remember changing this part, but the system is running for a couple of years already, so I might have done it…

[manjaro /]# lsblk -f
NAME        FSTYPE FSVER LABEL UUID FSAVAIL FSUSE% MOUNTPOINTS
loop0                                              
loop1                                              
loop2                                              
loop3                                              
sda                                                
|-sda1                                             
| |-ventoy                                         
| `-sda1                                           
`-sda2                                             
nvme0n1                                            
|-nvme0n1p1                          479.6M     6% /boot/efi
|-nvme0n1p2                          773.9M    14% /boot
|-nvme0n1p3                                        
| `-mydata                           306.6G    47% /
|-nvme0n1p4                                        
|-nvme0n1p5                                        
`-nvme0n1p6     
[manjaro /]# cat /etc/mkinitcpio.conf
# vim:set ft=sh
# MODULES
# The following modules are loaded before any boot hooks are
# run.  Advanced users may wish to specify all system modules
# in this array.  For instance:
#     MODULES=(usbhid xhci_hcd)
MODULES=()

# BINARIES
# This setting includes any additional binaries a given user may
# wish into the CPIO image.  This is run last, so it may be used to
# override the actual binaries included by a given hook
# BINARIES are dependency parsed, so you may safely ignore libraries
BINARIES=()

# FILES
# This setting is similar to BINARIES above, however, files are added
# as-is and are not parsed in any way.  This is useful for config files.
FILES=()

# HOOKS
# This is the most important setting in this file.  The HOOKS control the
# modules and scripts added to the image, and what happens at boot time.
# Order is important, and it is recommended that you do not change the
# order in which HOOKS are added.  Run 'mkinitcpio -H <hook name>' for
# help on a given hook.
# 'base' is _required_ unless you know precisely what you are doing.
# 'udev' is _required_ in order to automatically load modules
# 'filesystems' is _required_ unless you specify your fs modules in MODULES
# Examples:
##   This setup specifies all modules in the MODULES setting above.
##   No RAID, lvm2, or encrypted root is needed.
#    HOOKS=(base)
#
##   This setup will autodetect all modules for your system and should
##   work as a sane default
#    HOOKS=(base udev autodetect modconf block filesystems fsck)
#
##   This setup will generate a 'full' image which supports most systems.
##   No autodetection is done.
#    HOOKS=(base udev modconf block filesystems fsck)
#
##   This setup assembles a mdadm array with an encrypted root file system.
##   Note: See 'mkinitcpio -H mdadm_udev' for more information on RAID devices.
#    HOOKS=(base udev modconf keyboard keymap consolefont block mdadm_udev encrypt filesystems fsck)
#
##   This setup loads an lvm2 volume group.
#    HOOKS=(base udev modconf block lvm2 filesystems fsck)
#
##   This will create a systemd based initramfs which loads an encrypted root filesystem.
#    HOOKS=(base systemd autodetect modconf kms keyboard sd-vconsole sd-encrypt block filesystems fsck)
#
##   NOTE: If you have /usr on a separate partition, you MUST include the
#    usr and fsck hooks.
HOOKS=(base udev autodetect microcode modconf kms keyboard keymap consolefont block filesystems fsck)

# COMPRESSION
# Use this to compress the initramfs image. By default, zstd compression
# is used for Linux ≥ 5.9 and gzip compression is used for Linux < 5.9.
# Use 'cat' to create an uncompressed image.
#COMPRESSION="zstd"
#COMPRESSION="gzip"
#COMPRESSION="bzip2"
#COMPRESSION="lzma"
#COMPRESSION="xz"
#COMPRESSION="lzop"
#COMPRESSION="lz4"

# COMPRESSION_OPTIONS
# Additional options for the compressor
#COMPRESSION_OPTIONS=()

# MODULES_DECOMPRESS
# Decompress loadable kernel modules and their firmware during initramfs
# creation. Switch (yes/no).
# Enable to allow further decreasing image size when using high compression
# (e.g. xz -9e or zstd --long --ultra -22) at the expense of increased RAM usage
# at early boot.
# Note that any compressed files will be placed in the uncompressed early CPIO
# to avoid double compression.
#MODULES_DECOMPRESS="no"

I remember that I merged this file with a .pacnew file recently (I have ignored pacnew files completely until last week).
I tried to make sure that I don’t blindly overwrite the changes that were already made, but I might have overlooked something.
The system was booting just fine after the change, but maybe the recent update triggered something that broke the bootloader configuration.

Your system does not appear encrypted - you have no mapped luks partition - which may explain why you are having issues - your grub default points to a cryptdevice but none seems to exist.

Is the root the ‘mydata’ device?

|-nvme0n1p3                                        
| `-mydata                           306.6G    47% /

Huh? It definitely encrypted.
I had to open the encrypted partition using cryptsetup luksOpen /dev/nvme0n1p3 mydata before I was able to mount it.
Otherwise I was getting:

 mount /dev/nvme0n1p3 /mnt
mount: /mnt: unknown filesystem type 'crypto_LUKS'.
       dmesg(1) may have more information after failed mount system call.

So yes, root is mapped to mydata (or the other way around…?).

Yes - noted - I thought is was data - why is there no UUID listed ?

Is it in chroot ? I wanted to understand why the same UUID is listed three times in your grub default - look strange to me.

When I list my working system

 $ lsblk -f
NAME        FSTYPE      FSVER LABEL  UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
nvme0n1                                                                                  
├─nvme0n1p1 vfat        FAT32 EFI    B867-3C85                             376,7M    26% /efi
├─nvme0n1p2                                                                              
│ └─swap    swap        1     swap   e80c48d1-4ff4-4671-a8be-6d7c81690956                [SWAP]
└─nvme0n1p3 crypto_LUKS 2            a79fc6ed-c0ba-4c85-9301-88e273a19daf                
  └─system  btrfs             system 8f8a0469-0d6d-4f10-adb6-fb397e4afe01  362,4G    18% /home
                                                                                         /

All right, thanks for the explanation and your great help.
This actually did the trick, and I was able to reboot normally :partying_face:
The only thing that I am missing, is the Plymouth (?) UI/ prompt for the passphrase.
Right now, it just opens a command line after I select Manjaro in the boot menu and asks for it there.
Not a big thing, but it would be a bit nicer if there was the old UI again.