LUKS boot disk not unlocking at boot, but from live disk

TL;DR: My boot disk does not unlock from GRUB, but from a live disk, and chrooting into it and running mkinitcpio -P and update-grub did not help.

Background: I have updated all my LUKS headers to LUKS2 and the argon2id pbkdf, following this advice.

Afterwards, all LUKS devices unlock fine, both with their passphrases and their keyfiles. Including the boot disk, when unlocked from a live disk, but not when I try to boot into it. Must be GRUB, right?

On power on, it’s “Enter your passphrase…” => I do => “error: invalid passphrase” => “disk not found”.
The UUID of the disk is correct. The passphrase works from live system.

I unlocked the disk, chrooted into it and ran mkinitcpio -P, then update-grub. No change.

# mkinitcpio -P  &&  update-grub
==> Building image from preset: /etc/mkinitcpio.d/linux515.preset: 'default'
  -> -k /boot/vmlinuz-5.15-x86_64 -c /etc/mkinitcpio.conf -g /boot/initramfs-5.15-x86_64.img --microcode /boot/intel-ucode.img
==> Starting build: '5.15.106-1-MANJARO'
  -> Running build hook: [base]
  -> Running build hook: [udev]
  -> Running build hook: [autodetect]
  -> Running build hook: [modconf]
  -> Running build hook: [block]
  -> Running build hook: [keyboard]
  -> Running build hook: [keymap]
  -> Running build hook: [consolefont]
==> WARNING: consolefont: no font found in configuration
  -> Running build hook: [plymouth]
  -> Running build hook: [encrypt]
  -> Running build hook: [openswap]
  -> Running build hook: [resume]
  -> Running build hook: [filesystems]
==> Generating module dependencies
==> Creating gzip-compressed initcpio image: '/boot/initramfs-5.15-x86_64.img'
==> Image generation successful
==> Building image from preset: /etc/mkinitcpio.d/linux515.preset: 'fallback'
  -> -k /boot/vmlinuz-5.15-x86_64 -c /etc/mkinitcpio.conf -g /boot/initramfs-5.15-x86_64-fallback.img -S autodetect --microcode /boot/intel-ucode.img
==> Starting build: '5.15.106-1-MANJARO'
  -> Running build hook: [base]
  -> Running build hook: [udev]
  -> Running build hook: [modconf]
  -> Running build hook: [block]
==> WARNING: Possibly missing firmware for module: 'aic94xx'
==> WARNING: Possibly missing firmware for module: 'bfa'
==> WARNING: Possibly missing firmware for module: 'qed'
==> WARNING: Possibly missing firmware for module: 'qla1280'
==> WARNING: Possibly missing firmware for module: 'qla2xxx'
==> WARNING: Possibly missing firmware for module: 'wd719x'
  -> Running build hook: [keyboard]
  -> Running build hook: [keymap]
  -> Running build hook: [consolefont]
==> WARNING: consolefont: no font found in configuration
  -> Running build hook: [plymouth]
  -> Running build hook: [encrypt]
  -> Running build hook: [openswap]
  -> Running build hook: [resume]
  -> Running build hook: [filesystems]
==> Generating module dependencies
==> Creating gzip-compressed initcpio image: '/boot/initramfs-5.15-x86_64-fallback.img'
==> Image generation successful
==> Building image from preset: /etc/mkinitcpio.d/linux61.preset: 'default'
  -> -k /boot/vmlinuz-6.1-x86_64 -c /etc/mkinitcpio.conf -g /boot/initramfs-6.1-x86_64.img --microcode /boot/intel-ucode.img
==> Starting build: '6.1.23-1-MANJARO'
  -> Running build hook: [base]
  -> Running build hook: [udev]
  -> Running build hook: [autodetect]
  -> Running build hook: [modconf]
  -> Running build hook: [block]
==> WARNING: Possibly missing firmware for module: 'xhci_pci'
  -> Running build hook: [keyboard]
  -> Running build hook: [keymap]
  -> Running build hook: [consolefont]
==> WARNING: consolefont: no font found in configuration
  -> Running build hook: [plymouth]
  -> Running build hook: [encrypt]
  -> Running build hook: [openswap]
  -> Running build hook: [resume]
  -> Running build hook: [filesystems]
==> Generating module dependencies
==> Creating gzip-compressed initcpio image: '/boot/initramfs-6.1-x86_64.img'
==> Image generation successful
==> Building image from preset: /etc/mkinitcpio.d/linux61.preset: 'fallback'
  -> -k /boot/vmlinuz-6.1-x86_64 -c /etc/mkinitcpio.conf -g /boot/initramfs-6.1-x86_64-fallback.img -S autodetect --microcode /boot/intel-ucode.img
==> Starting build: '6.1.23-1-MANJARO'
  -> Running build hook: [base]
  -> Running build hook: [udev]
  -> Running build hook: [modconf]
  -> Running build hook: [block]
==> WARNING: Possibly missing firmware for module: 'aic94xx'
==> WARNING: Possibly missing firmware for module: 'bfa'
==> WARNING: Possibly missing firmware for module: 'qed'
==> WARNING: Possibly missing firmware for module: 'qla1280'
==> WARNING: Possibly missing firmware for module: 'qla2xxx'
==> WARNING: Possibly missing firmware for module: 'wd719x'
==> WARNING: Possibly missing firmware for module: 'xhci_pci'
  -> Running build hook: [keyboard]
  -> Running build hook: [keymap]
  -> Running build hook: [consolefont]
==> WARNING: consolefont: no font found in configuration
  -> Running build hook: [plymouth]
  -> Running build hook: [encrypt]
  -> Running build hook: [openswap]
  -> Running build hook: [resume]
  -> Running build hook: [filesystems]
==> Generating module dependencies
==> Creating gzip-compressed initcpio image: '/boot/initramfs-6.1-x86_64-fallback.img'
==> Image generation successful
Generating grub configuration file ...
Found theme: /usr/share/grub/themes/manjaro/theme.txt
Found linux image: /boot/vmlinuz-6.1-x86_64
Found initrd image: /boot/intel-ucode.img /boot/initramfs-6.1-x86_64.img
Found initrd fallback image: /boot/initramfs-6.1-x86_64-fallback.img
Found linux image: /boot/vmlinuz-5.15-x86_64
Found initrd image: /boot/intel-ucode.img /boot/initramfs-5.15-x86_64.img
Found initrd fallback image: /boot/initramfs-5.15-x86_64-fallback.img
Warning: os-prober will not be executed to detect other bootable partitions.
Systems on them will not be added to the GRUB boot configuration.
Check GRUB_DISABLE_OS_PROBER documentation entry.
Adding boot menu entry for UEFI Firmware Settings ...
Detecting snapshots ...
Found snapshot: 2023-04-20 13:55:55 | timeshift-btrfs/snapshots/2023-04-20_13-55-55/@ | ondemand                             | {timeshift-autosnap} {created before upgrade} |

<snip>

Found snapshot: 2023-02-22 15:08:43 | timeshift-btrfs/snapshots/2023-02-22_15-08-43/@ | ondemand hourly daily weekly monthly | fresh install autosnap before upgrade         |
Found 50 snapshot(s)
Unmount /tmp/grub-btrfs.TA9NcKvavf .. Success
Found memtest86+ image: /boot/memtest86+/memtest.bin
Found memtest86+ EFI image: /boot/memtest86+/memtest.efi
done

I can see the /boot/initramfs-* that updated “just now”.

/etc/default/grub seems ok.

GRUB_DEFAULT=0
GRUB_TIMEOUT=1
GRUB_TIMEOUT_STYLE=hidden
GRUB_DISTRIBUTOR="Manjaro"
GRUB_CMDLINE_LINUX_DEFAULT="cryptdevice=UUID=553d95f9-e962-49ed-b6e7-bee7726a5ec1:luks-553d95f9-e962-49ed-b6e7-bee7726a5ec1 root=/dev/mapper/luks-553d95f9-e962-49ed-b6e7-bee7726a5ec1 resume=/dev/mapper/luks-13f9c137-c427-4be0-b567-33a209509780 udev.log_priority=3 nvme.noacpi=1 mem_sleep_default=deep"
GRUB_CMDLINE_LINUX=""


# If you want to enable the save default function, uncomment the following
# line, and set GRUB_DEFAULT to saved.
#GRUB_SAVEDEFAULT="true"

# Uncomment to disable submenus in boot menu
#GRUB_DISABLE_SUBMENU=y

# Preload both GPT and MBR modules so that they are not missed
GRUB_PRELOAD_MODULES="part_gpt part_msdos"

# Uncomment to enable booting from LUKS encrypted devices
GRUB_ENABLE_CRYPTODISK=y

# Uncomment to use basic console
GRUB_TERMINAL_INPUT=console

# Uncomment to disable graphical terminal
#GRUB_TERMINAL_OUTPUT=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command 'videoinfo'
GRUB_GFXMODE=auto

# Uncomment to allow the kernel use the same resolution used by grub
GRUB_GFXPAYLOAD_LINUX=keep

# Uncomment if you want GRUB to pass to the Linux kernel the old parameter
# format "root=/dev/xxx" instead of "root=/dev/disk/by-uuid/xxx"
#GRUB_DISABLE_LINUX_UUID=true

# Uncomment to disable generation of recovery mode menu entries
GRUB_DISABLE_RECOVERY=true

# Uncomment this option to enable os-prober execution in the grub-mkconfig command
#GRUB_DISABLE_OS_PROBER=false

# Uncomment and set to the desired menu colors.  Used by normal and wallpaper
# modes only.  Entries specified as foreground/background.
GRUB_COLOR_NORMAL="light-gray/black"
GRUB_COLOR_HIGHLIGHT="green/black"

# Uncomment one of them for the gfx desired, a image background or a gfxtheme
#GRUB_BACKGROUND="/usr/share/grub/background.png"
GRUB_THEME="/usr/share/grub/themes/manjaro/theme.txt"

# Uncomment to get a beep at GRUB start
#GRUB_INIT_TUNE="480 440 1"

# Uncomment to ensure that the root filesystem is mounted read-only so that
# systemd-fsck can run the check automatically. We use 'fsck' by default, which
# needs 'rw' as boot parameter, to avoid delay in boot-time. 'fsck' needs to be
# removed from 'mkinitcpio.conf' to make 'systemd-fsck' work.
# See also Arch-Wiki: https://wiki.archlinux.org/index.php/Fsck#Boot_time_checking
#GRUB_ROOT_FS_RO=true

GRUB_ENABLE_CRYPTODISK=y

Check the UUIDS:

# blkid  | grep LUKS
/dev/nvme0n1p3: UUID="13f9c137-c427-4be0-b567-33a209509780" TYPE="crypto_LUKS" PARTUUID="89d7b623-3e8a-d749-a3c7-3d65c9bb5149"
/dev/nvme0n1p2: UUID="553d95f9-e962-49ed-b6e7-bee7726a5ec1" TYPE="crypto_LUKS" PARTLABEL="root" PARTUUID="4bd28b0f-624a-574d-bc5d-eea402fcccdf"

( …5ec1 is encrypted root, …9780 is encrypted swap)

Relevant crypttab entries are:

# <name>                                  <device>                                   <password>                         <options>

# internal SSD
luks-553d95f9-e962-49ed-b6e7-bee7726a5ec1 UUID=553d95f9-e962-49ed-b6e7-bee7726a5ec1  /crypto_keyfile.bin                luks
luks-13f9c137-c427-4be0-b567-33a209509780 UUID=13f9c137-c427-4be0-b567-33a209509780  /crypto_keyfile.bin                luks

<snip>

Relevant fstab entries are:

# internal SSD
UUID=71A5-1BE9                                          /boot/efi                             vfat    umask=0077,noatime                                                                              0 2
/dev/mapper/luks-553d95f9-e962-49ed-b6e7-bee7726a5ec1   /                                     btrfs   subvol=/@,discard=async,ssd,noatime                                                             0 0
/dev/mapper/luks-553d95f9-e962-49ed-b6e7-bee7726a5ec1   /home                                 btrfs   subvol=/@home,discard=async,ssd,noatime                                                         0 0
/dev/mapper/luks-553d95f9-e962-49ed-b6e7-bee7726a5ec1   /var/cache                            btrfs   subvol=/@cache,discard=async,ssd,noatime                                                        0 0
/dev/mapper/luks-553d95f9-e962-49ed-b6e7-bee7726a5ec1   /var/log                              btrfs   subvol=/@log,discard=async,ssd,noatime                                                          0 0
/dev/mapper/luks-13f9c137-c427-4be0-b567-33a209509780   swap                                  swap    noatime                                                                                         0 0
tmpfs                                                   /tmp                                  tmpfs   noatime,mode=1777                                                                               0 0

And /etc/mkinitcpio.conf reads:

# vim:set ft=sh
# MODULES
# The following modules are loaded before any boot hooks are
# run.  Advanced users may wish to specify all system modules
# in this array.  For instance:
#     MODULES=(usbhid xhci_hcd)
MODULES="crc32c-intel"

# BINARIES
# This setting includes any additional binaries a given user may
# wish into the CPIO image.  This is run last, so it may be used to
# override the actual binaries included by a given hook
# BINARIES are dependency parsed, so you may safely ignore libraries
BINARIES=()

# FILES
# This setting is similar to BINARIES above, however, files are added
# as-is and are not parsed in any way.  This is useful for config files.
FILES="/crypto_keyfile.bin"

# HOOKS
# This is the most important setting in this file.  The HOOKS control the
# modules and scripts added to the image, and what happens at boot time.
# Order is important, and it is recommended that you do not change the
# order in which HOOKS are added.  Run 'mkinitcpio -H <hook name>' for
# help on a given hook.
# 'base' is _required_ unless you know precisely what you are doing.
# 'udev' is _required_ in order to automatically load modules
# 'filesystems' is _required_ unless you specify your fs modules in MODULES
# Examples:
##   This setup specifies all modules in the MODULES setting above.
##   No RAID, lvm2, or encrypted root is needed.
#    HOOKS=(base)
#
##   This setup will autodetect all modules for your system and should
##   work as a sane default
#    HOOKS=(base udev autodetect modconf block filesystems fsck)
#
##   This setup will generate a 'full' image which supports most systems.
##   No autodetection is done.
#    HOOKS=(base udev modconf block filesystems fsck)
#
##   This setup assembles a mdadm array with an encrypted root file system.
##   Note: See 'mkinitcpio -H mdadm_udev' for more information on RAID devices.
#    HOOKS=(base udev modconf keyboard keymap consolefont block mdadm_udev encrypt filesystems fsck)
#
##   This setup loads an lvm2 volume group.
#    HOOKS=(base udev modconf block lvm2 filesystems fsck)
#
##   NOTE: If you have /usr on a separate partition, you MUST include the
#    usr and fsck hooks.
HOOKS="base udev autodetect modconf block keyboard keymap consolefont plymouth encrypt openswap resume filesystems"

# COMPRESSION
# Use this to compress the initramfs image. By default, gzip compression
# is used. Use 'cat' to create an uncompressed image.
#COMPRESSION="gzip"
#COMPRESSION="bzip2"
#COMPRESSION="lzma"
#COMPRESSION="xz"
#COMPRESSION="lzop"
#COMPRESSION="lz4"
#COMPRESSION="zstd"

# COMPRESSION_OPTIONS
# Additional options for the compressor
#COMPRESSION_OPTIONS=()

# MODULES_DECOMPRESS
# Decompress kernel modules during initramfs creation.
# Enable to speedup boot process, disable to save RAM
# during early userspace. Switch (yes/no).
#MODULES_DECOMPRESS="yes"

Now, I have header backups, but they are still LUKS1. Soooo close to bang my head against a wall…

Your /boot is unencrypted right?

(Also, if you include the crypto_keyfile.bin in your unencrypted boot, this seems rather useless then using a strong pbkdf, no?)

Did you upgrade grub (actually upgrade the installation by installing it again)?
It might require a newer kernel, you could test with this.

You could also try the systemd-hooks rather than the old (?) ones.

HOOKS=(base systemd sd-vconsole sd-plymouth keyboard autodetect modconf kms block sd-encrypt lvm2 filesystems)

(if you don’t have plymouth, remove this hook)

Also, if you don’t require grub, you could also switch to systemd-boot.

Most likely the issue, Last I checked GRUB needed pbkdf2.

Ah yes, right, this is it:

From your linked post: :cry:

Also, if you’re using an encrypted /boot, stop now - very recent versions of grub2 support LUKS2, but they don’t support argon2id, and this will render your system unbootable.

(It works with the systemd-hooks and systemd-boot.)

It’s copied out for testing purposes. The flash stick will be dded over afterwards, and nobody but me has physical access.

Not yet. Good point.

Aaargh! Thank you so much!

I think that is the kick to switch to systemd; no necessity to stay with GRUB.

We have a saying here – not really proverb but heard frequently – “Wer lesen kann, ist klar im Vorteil.” (“[S]he who can read clearly has the advantage.”) Sooo true.

Thank you, guys!