Architect install - boot fails on encrypted volumes

[Jade1]

Mod-edit: Formatting

---

Hi,

I installed manjaro using the ‘Architect install’.
I followed the excellent guide by @davidde:

[HowTo] Install Manjaro fully encrypted using Manjaro Architect

Difficulty: ★★★★☆ We will be creating an LVM on LUKS encrypted Manjaro installation with UEFI and GPT using Manjaro Architect. This can currently be considered the happy medium between complete Full Disk Encryption and usability. While technically not FDE, both / and /home are fully encrypted (with the exception of a small 0.5GiB /boot partition), but at the same time, boot times are great and complexity is minimal. Make sure to boot the Manjaro-Architect iso in UEFI mode, otherwise the GRUB U…

The reason I went down the Architect install route was because I got a new laptop with 1 nvme drive but I added a 2nd nvme drive after I received it.
None of the graphical install options gave me the possibility of configuring the 2nd nvme drive during the install process so I resorted to the Architect install.

On the 1st SSD, I have a non-lvm, non-encrypted partition for /boot.
The rest of the 1st SSD is partitioned with LVM.
The 2nd SSD is partitioned with LVM.

I have 3 encrypted logical volumes (the swap partition is also encrypted).
The install process was smooth and I followed all the steps and it completed successfully.

When I rebooted the system, it gets stuck on:

A start job is running for /dev/disk/by-uuid/<uuid> (*n* min *n* s / 1min 30s)

It toggles the uuid for each of the 3 encrypted logical volumes and timer countsdown until it gets to 1min 30s and then it goes into emergency state.

I never get prompted for the password(s) to decrypt the lvms.

There are no entries in my /etc/crypttab but there are entries in my /etc/fstab.
It seems like I may have inadvertently missed a step but I’m not sure what…

Any suggestions to help resolve are appreciated.

TIA

[TriMoon]

My first assumption would be that you have forgotten to add the encrypt or sd-encrypt hooks, depending on what you need, in your /etc/mkinitcpio.conf

Can you check out 👩‍🏫[HowTo] Boot without a password for encrypted root partition, it has some nice info you might also be interested in…
Note to self: Need to add that info in my how-to also…

[Jade1]

Thanks for the very detailed instructions! I have read through and trying to work through it. I am trying to enable efi.automount and I am getting errors:

“efi.mount: What= setting is missing. Refusing.”
“efi.automount: Refusing to start, unit efi.mount to trigger not loaded.”
“Failed to set up automount ESP.”

I am using systemd-boot as the boot loader. On my system, there is nothing under /dev/disk/by-label/ but I do have a few entries under /dev/disk/by-uuid. Should I be using /dev/disk/by-uuid instead under “What” in /etc/systemd/system/efi.mount.d/hd.conf?

[Jade1]

I have added the sd-encrypt hooks to my /etc/mkinitcpio.conf and updated using mkinitcpio -p. I notice though that the password details do not get copied to /etc/crypttab by the mkinitcpio command.

After rebooting, the system still gets stuck on trying to decrypt the 3 lvols. However, after it waits for 1min 30s, it now continues to boot to the graphical environment. I am not able to login though as my home directory is encrypted and it was never decrypted during the boot process.

I’ll look through my configuration again to see what I may be missing…

[TriMoon]

@Jade1

“efi.mount: What= setting is missing. Refusing.”
“efi.automount: Refusing to start, unit efi.mount to trigger not loaded.”
“Failed to set up automount ESP.”

The “What=” config is given in the override /etc/systemd/system/efi.mount.d/hd.conf
I did that to be able to use different settings, one for HD and one for the USB, by doing a simple rename of one or the other with a dot in-front
Did you reload the daemon after you added those units/files?

@Jade1

I am using systemd-boot as the boot loader.

I do also…(click my avatar to see it mentioned)

@Jade1

On my system, there is nothing under /dev/disk/by-label/ but I do have a few entries under /dev/disk/by-uuid. Should I be using /dev/disk/by-uuid instead under “What” in /etc/systemd/system/efi.mount.d/hd.conf?

Yes because i’ve given my ESP a label like that, so you will need to do that also, or use it’s UUID/PARTUUID
(I used a volume label to make it more robust between installs)

@Jade1

I’ll look through my configuration again to see what I may be missing…

So will I, to see if i missed anything that needs to be added…
I didn’t notice anything i missed…

@Jade1

I have added the sd-encrypt hooks to my /etc/mkinitcpio.conf and updated using mkinitcpio -p. I notice though that the password details do not get copied to /etc/crypttab by the mkinitcpio command.

You do realize that the /etc/crypttab inside the initrd is a renamed /etc/crypttab.initramfs from your HD right?

[TriMoon]

PS:
I just added this piece, which might be the cause of the passwords not being recognized:

@TriMoon

Make sure the contents of these files don’t include a newline at end, because that will be provided as part of the password also !

[ion]

Hi, a couple of comments if I may:

@Jade1

On the 1st SSD, I have a non-lvm, non-encrypted partition for /boot.
The rest of the 1st SSD is partitioned with LVM.
The 2nd SSD is partitioned with LVM.

This double-SSD LVM makes your setup basically not fully compatible with the HowTo instructions you quote (because you create a LVM first), and also not TriMoon’s (which uses no LVM). Which HowTo do you try to implement at current?

@Jade1

I have added the sd-encrypt hooks to my /etc/mkinitcpio.conf

If you use sd-encrypt you need sd-lvm2 (instead of lvm2 for the LVM. See the following for cross-check: https://wiki.archlinux.org/index.php/Dm-crypt/Encrypting_an_entire_system#LUKS_on_LVM

[TriMoon]

@ion

and also not TriMoon’s (which uses no LVM)

My howto is generic anough to apply for all kinds of setup…
In fact my own setup config uses full disk encryption with an LVM inside, instead of an LVM with a luks container inside that
(See addendum with my mkinitcpio config)

His setup just doesn’t auto decrypt and doesn’t ask for password as far as i could tell…

[Jade1]

Thanks for all the great feedback TriMoon and ion. Much appreciated. I may have bitten off more than I can chew here!

I checked and there are no newlines in the files containing my passwords for the encrypted lvols:

/etc/cryptfs-keys.d/corporate
/etc/cryptfs-keys.d/john
/etc/cryptfs-keys.d/swap

The only reason I went for LUKS on LVM was because I thought there was a big penalty for having FDE and I only needed particular information encrypted (home, swap, company data). Perhaps the hit is not as big as I’m thinking it would be (I have a i7-10875H based laptop) and maybe save myself the headache and go for FDE so I can completely follow TriMoon’s guide?

I’ll need to review… The contents of /etc/mkinitcpio.conf, efi.automount, efi.mount.d/hd.conf, boot.mount, boot.automount is exactly the same as TriMoon’s. What I am not entirely clear about though is what I need to put in /etc/systemd/system/efi.mount.d/hd.conf. I don’t think I can use the example file provided by TriMoon verbatim, but I’m not sure what I need to substitute based on my setup…

Output from lsblk --fs:

nvme0n1                                                                                                                                  
    └─nvme0n1p1                 LVM2_member LVM2 001                                   U2ff8Q-BLP0-FHBv-b9Pj-WZOK-Wpr3-3aG5T2                
      ├─vg02-lvol_lxd           ext4        1.0                                        4fea769e-abc4-4fcb-8208-6e7d17c1c950     46.4G     0% /mnt/var/lib/lxd
      ├─vg02-lvol_downloads     ext4        1.0                                        1efeee99-d440-47ee-8009-76f88430a60f                  
      ├─vg02-lvol_install_discs ext4        1.0                                        044d7519-8473-4b9a-9f0e-43bf913345e1     46.4G     0% /mnt/installDiscs
      ├─vg02-lvol_libvirt       ext4        1.0                                        0f7ad6ea-4cbc-4c75-90f2-6ace7aba66ea    139.1G     0% /mnt/var/lib/libvirt
      └─vg02-lvol_swap          crypto_LUKS 1                                          31906923-c04f-49e0-9dfd-74bcd6435475                  
    nvme1n1                                                                                                                                  
    ├─nvme1n1p1                 vfat        FAT32                                      DBB9-D8DF                               281.1M    45% /mnt/boot
    └─nvme1n1p2                 LVM2_member LVM2 001                                   tcYhjG-vf2H-Lhxl-SvNb-dGw6-qbqZ-27q4UK                
      ├─vg01-lvol_home          ext4        1.0                                        29f47f99-d263-4076-b2a1-6b990c3a35e6      9.2G     1% /mnt/home
      ├─vg01-lvol_john          crypto_LUKS 1                                          5ba2e072-c74b-45f4-8e4e-4feae29b2e50                  
      ├─vg01-lvol_root          ext4        1.0                                        a319cae0-246f-46a3-a2d7-8c9e2fb6a49a     82.7G    10% /mnt
      └─vg01-lvol_corporate     crypto_LUKS 1                                          c6563ef6-38a5-49ab-95cf-6158bb49ffbb

Contents of crypttab.initramfs:

vg02-lvol_swap        UUID=31906923-c04f-49e0-9dfd-74bcd6435475    /etc/cryptfs-keys.d/swap
vg01-lvol_john        UUID=5ba2e072-c74b-45f4-8e4e-4feae29b2e50    /etc/cryptfs-keys.d/john
vg01-lvol_corporate   UUID=c6563ef6-38a5-49ab-95cf-6158bb49ffbb    /etc/cryptfs-keys.d/corporate

/dev/disk/by-uuid

...
31906923-c04f-49e0-9dfd-74bcd6435475
5ba2e072-c74b-45f4-8e4e-4feae29b2e50
c6563ef6-38a5-49ab-95cf-6158bb49ffbb
...

/etc/fstab

# /dev/mapper/vg01-lvol_root
UUID=a319cae0-246f-46a3-a2d7-8c9e2fb6a49a	/         	ext4      	rw,noatime	0 0

# /dev/mapper/vg01-lvol_home
UUID=29f47f99-d263-4076-b2a1-6b990c3a35e6	/home     	ext4      	rw,noatime	0 0

# /dev/mapper/cryptjohn
UUID=d410f2bb-ed79-4b46-9413-3c3fa9da82bd	/home/john	ext4      	rw,noatime	0 0

# /dev/mapper/vg02-lvol_downloads
UUID=1efeee99-d440-47ee-8009-76f88430a60f	/home/john/Downloads	ext4      	rw,noatime	0 0

# /dev/mapper/crypt_corporate
UUID=1ffd86e7-0d7c-4308-b055-ed41e5a9f979	/corporate	ext4      	rw,noatime	0 0

# /dev/mapper/vg02-lvol_lxd
UUID=4fea769e-abc4-4fcb-8208-6e7d17c1c950	/var/lib/lxd	ext4      	rw,noatime	0 0

# /dev/mapper/vg02-lvol_install_discs
UUID=044d7519-8473-4b9a-9f0e-43bf913345e1	/installDiscs	ext4      	rw,noatime	0 0

# /dev/mapper/vg02-lvol_libvirt
UUID=0f7ad6ea-4cbc-4c75-90f2-6ace7aba66ea	/var/lib/libvirt	ext4      	rw,noatime	0 0

# /dev/nvme1n1p1
UUID=DBB9-D8DF      	/boot     	vfat      	rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,utf8,errors=remount-ro	0 0

# /dev/mapper/crypt_swap
UUID=5474d6ab-a6bb-4a93-9ecb-dbed716ec0bd	none      	swap      	defaults,pri=-2	0 0

As for the UUIDs in my crypttab.initramfs, I got these from the output of lsblk --fs. I notice though that the UUIDs for these are different to those in my /etc/fstab. Should the UUIDs be the same?

[TriMoon]

@Jade1

What I am not entirely clear about though is what I need to put in /etc/systemd/system/efi.mount.d/hd.conf

Where is your ESP located?
If your ESP has a volume label that reads “ESP” you don’t need to change anything.
Otherwise you need to use your volume label instead of “ESP” in that config.
If it has no volume label then you need to change it, so it points to the partuuid of your ESP.

@Jade1

Contents of crypttab.initramfs:

vg02-lvol_swap        UUID=31906923-c04f-49e0-9dfd-74bcd6435475    /etc/cryptfs-keys.d/swap
vg01-lvol_john        UUID=5ba2e072-c74b-45f4-8e4e-4feae29b2e50    /etc/cryptfs-keys.d/john
vg01-lvol_corporate   UUID=c6563ef6-38a5-49ab-95cf-6158bb49ffbb    /etc/cryptfs-keys.d/corporate

You can’t use the container names of your LV’s here like that because you need to use different names.
See man crypttab
Names used in there are container names created to access the contents of the encrypted partitions, so you should use names like:

• luks_swap (will become accessible as /dev/mapper/luks_swap)
• enc_john
• corp_sensitive_data

etc, and use those names to mount those in your mount logic, eg. fstab

wrt: your fstab: You should not use UUID’s there, instead use the full device mapper path that are created by the above.

Eg, what your setup as posted uses is:

A disk
└─ An LVM
   └─ An LV
      └─ An encrypted container.  (Accessible as UUID or /dev/mapper/some_lv_name)
         └─ The decrypted data. (Accessible only as /dev/mapper/some_name_from_crypttab)

[Jade1]

Thanks for sticking with me @TriMoon! I made it!!

I now have a system which boots into a graphical environment with all my encrypted and non-encrypted volumes mounted!

I made changes to the following files:

crypttab.initramfs

luks_swap        UUID=31906923-c04f-49e0-9dfd-74bcd6435475    /etc/cryptfs-keys.d/swap
luks_john        UUID=5ba2e072-c74b-45f4-8e4e-4feae29b2e50    /etc/cryptfs-keys.d/john
luks_corporate   UUID=c6563ef6-38a5-49ab-95cf-6158bb49ffbb    /etc/cryptfs-keys.d/corporate

/etc/fstab

...
#/dev/mapper/cryptjohn
/dev/mapper/luks_john	/home/john	ext4      	rw,noatime	0 0
...
# /dev/mapper/crypt_corporate
/dev/mapper/luks_corporate	/corporate	ext4      	rw,noatime	0 0
...
# /dev/mapper/crypt_swap
/dev/mapper/luks_swap	none      	swap      	defaults,pri=-2	0 0

For this next one, I wish I’d have figured out sooner that \x2d is of course the hex representation of “-”. For future readers, DBB9-D8Df is the UUID for my ESP from /dev/disk/by-uuid.

/etc/systemd/system/efi.mount.d/hd.conf

[Unit]
Description=ESP (HD)
After=blockdev@dev-disk-by\x2duuid-DBB9\x2dD8DF.target
BindsTo=dev-disk-by\x2duuid-DBB9\x2dD8DF.device

# Do a fsck on the partition
Requires=systemd-fsck@dev-disk-by\x2duuid-DBB9\x2dD8DF.service
After=systemd-fsck@dev-disk-by\x2duuid-DBB9\x2dD8DF.service

[Mount]
What=/dev/disk/by-uuid/DBB9-D8DF

The only outstanding item is that I still have to enter my password 3 times during the boot process to decrypt the 3 encrypted volumes. I’m not sure why the passwords are not being taken from /etc/cryptfs-keys.d

[TriMoon]

@Jade1

I checked and there are no newlines in the files containing my passwords for the encrypted lvols:

@Jade1

The only outstanding item is that I still have to enter my password 3 times during the boot process to decrypt the 3 encrypted volumes. I’m not sure why the passwords are not being taken from /etc/cryptfs-keys.d

Did you put the passwords in those files using printf “%s” “somepassword” > yourpasswordfile?
Because if not you might have an un-intended newline which might be hard to notice
Also check the contents of the generated initrd if it actually has the correct files and contents with respect to this. Eg crypttab and password files.

@Jade1

For this next one, I wish I’d have figured out sooner that \x2d is of course the hex representation of “-”.

Use systemd-escape for the translation in future

[Jade1]

@TriMoon

Did you put the passwords in those files using printf “%s” “somepassword” > yourpasswordfile?

No. I just “vi-ed” the file. Using hexdump, I saw that there was indeed a “0a” at the end of the line that wasn’t visible to the naked eye! Thanks for saving my bacon yet again!

I ran mkinitcpio -P again and presto, I’m no longer prompted for a password! For all those times I had to run mkinitcpio, I had to reboot, drop into a “live iso” session, run “manjaro-chroot -a” and then mkinitcpio. I presume that it’s not possible to run mkinitcpio from a booted system? I tried but the command just hangs.

@TriMoon

Use systemd-escape for the translation in future

Nice tip - thanks! I never knew about that functionality.

[TriMoon]

I’m glad to hear you got it working as intended now.

@Jade1

I presume that it’s not possible to run mkinitcpio from a booted system?

If that was not possible it would mean every time you install something that runs that command automatically, it would make your system unable to start

You only need to use the chroot when you first setup your system so it operates on the correct files.

@Jade1

@TriMoon

Did you put the passwords in those files using printf “%s” “somepassword” > yourpasswordfile?

No. I just “vi- ed ” the file. Using hexdump, I saw that there was indeed a “0a” at the end of the line that wasn’t visible to the naked eye! Thanks for saving my bacon yet again!

Added that tip to the howto also

[TriMoon]

@TriMoon

         └─ The decrypted data. (Accessible only as /dev/mapper/some_name_from_crypttab)

Correction to what i said there:

• That seems to be accessible by UUID also as i found out recently:
  My switch to the 5.10 kernel with update 2020-12-30

[Jade1]

Sadly, I’ve managed to break my system! I executed pacman -Syu, rebooted and I’m now back to getting stuck at
A start job is running for /dev/disk/by-uuid/<uuid> (*n* min *n* s / 1min 30s)

I’ve tried to run mkinitspio -P but I get errors on:
==> ERROR: ‘/lib/modules/5.4.80-2-MANJARO’ is not a valid kernel module directory

Output of /lib/modules

[manjaro /]# ls -l /lib/modules
total 20
drwxr-xr-x 4 root root 4096 Dec 31 11:58 5.4.85-1-MANJARO
drwxr-xr-x 4 root root 4096 Dec 27 09:02 5.9.11-3-MANJARO
drwxr-xr-x 4 root root 4096 Dec 31 11:59 5.9.16-1-MANJARO
drwxr-xr-x 2 root root 4096 Dec 31 11:58 extramodules-5.4-MANJARO
drwxr-xr-x 2 root root 4096 Dec 31 11:58 extramodules-5.9-MANJARO

I think I remember reading somewhere that I needed to do something extra if using systemd-boot to ensure no issues with updates. I’ll look into this…

[TriMoon]

That error in combination with the directory listing suggests you have a lingering older 5.4 kernel install somewhere…
But anyhow recap what you did to break your no-password boot in first place, it might be related

[Jade1]

I couldn’t get to the root cause of what broke my setup…I decided to re-install to give myself some practice as well!

I now have an issue with the filesystem type for my ESP. I’m not sure what I’ve done differently with this install

Output of systemctl status efi.mount

efi.mount - ESP (HD)
     Loaded: loaded (/etc/systemd/system/efi.mount; static)
    Drop-In: /etc/systemd/system/efi.mount.d
             └─hd.conf
     Active: failed (Result: exit-code) since Sat 2021-01-02 01:46:42 EST; 2min 32s ago
TriggeredBy: ● efi.automount
      Where: /efi
       What: /dev/disk/by-uuid/413C-67AE
       Docs: man:systemd.mount(5)
             man:systemd.automount(5)

Jan 02 01:46:42 ja-clevo systemd[1]: Mounting ESP (HD)...
Jan 02 01:46:42 ja-clevo systemd[1]: efi.mount: Mount process exited, code=exited, status=32/n/a
Jan 02 01:46:42 ja-clevo mount[1138]: mount: /efi: wrong fs type, bad option, bad superblock on /dev/nvme1n1p1, missing codepage or helper program, or other error.
Jan 02 01:46:42 ja-clevo systemd[1]: efi.mount: Failed with result 'exit-code'.
Jan 02 01:46:42 ja-clevo mount[1139]: mount: /efi: wrong fs type, bad option, bad superblock on /dev/nvme1n1p1, missing codepage or helper program, or other error.
Jan 02 01:46:42 ja-clevo systemd[1]: Failed to mount ESP (HD).
Jan 02 01:46:42 ja-clevo systemd[1]: Mounting ESP (HD)...
Jan 02 01:46:42 ja-clevo systemd[1]: efi.mount: Mount process exited, code=exited, status=32/n/a
Jan 02 01:46:42 ja-clevo systemd[1]: efi.mount: Failed with result 'exit-code'.
Jan 02 01:46:42 ja-clevo systemd[1]: Failed to mount ESP (HD)

fstab

    ...
    # /dev/nvme1n1p1
UUID=413C-67AE          /boot           vfat            rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,utf8,errors=remount-ro       0 0
    ...

Output of bootctl

System:
     Firmware: UEFI 2.70 (INSYDE Corp. 24579.12295)
  Secure Boot: disabled
   Setup Mode: setup
 Boot into FW: supported

Current Boot Loader:
      Product: systemd-boot 247.2-1-manjaro
     Features: ✓ Boot counting
               ✓ Menu timeout control
               ✓ One-shot menu timeout control
               ✓ Default entry control
               ✓ One-shot entry control
               ✓ Support for XBOOTLDR partition
               ✓ Support for passing random seed to OS
               ✓ Boot loader sets ESP partition information
WARNING: The boot loader reports different ESP UUID then detected!
          ESP: /dev/disk/by-partuuid/1f6ffee3-93ec-4fef-b9f4-61e93eae4bb6
         File: └─/EFI/systemd/systemd-bootx64.efi

Random Seed:
 Passed to OS: yes
 System Token: set

Boot Loaders Listed in EFI Variables:
        Title: Linux Boot Manager
           ID: 0x0000
       Status: active, boot-order
    Partition: /dev/disk/by-partuuid/1f6ffee3-93ec-4fef-b9f4-61e93eae4bb6
         File: └─/EFI/systemd/systemd-bootx64.efi

[TriMoon]

@TriMoon

This special purpose partition (ESP) is meant to hold all bootloaders for your system and must be formatted as a FAT file system(UEFI-Specs §12.3 File System Format) (eg. fat32(vfat) ) without encryption.

if that is correct then:

@Jade1

Jan 02 01:46:42 ja-clevo mount[1138]: mount: /efi: wrong fs type, bad option, bad superblock on /dev/nvme1n1p1, missing codepage or helper program, or other error.

@Jade1

    # /dev/nvme1n1p1
UUID=413C-67AE          /boot           vfat            rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,utf8,errors=remount-ro       0 0

Suggests you are trying to load the ESP both on /efi and /boot at same time…

@Jade1

    Partition: /dev/disk/by-partuuid/1f6ffee3-93ec-4fef-b9f4-61e93eae4bb6

This shows the partuuid that the system is using as ESP…

[Jade1]

@TriMoon

Suggests you are trying to load the ESP both on /efi and /boot at same time…

I’m not sure how I managed to achieve that!