Errors during boot process!

Hi everyone,

I currently trying to get Manjaro Gnome to Install on a ZFS partitioned disk. I have a fat formatted EFI partition, a ZFS formatted boot partition and a ZFS formatted and nativly encrypted root parition. I setup the partitions and then install from the CLI using the setup command to start the architect like installer.

During boot after the installation I get error messages:

The boot until the mounting of the root partition seems to work fine, since at the point were it asks me for my password for root GRUB must have done it’s thing.
The error messages aren’t really googleable since the search does not turn up anything that seems related and I can’t really tell where my error in this process is since I do not really now what these messages mean.

Any input on the topic is greatly appreciated!

If this is of any concern this is what I do during the install process:

# create a short handle to reference the ssd by id
DISK=/dev/disk/by-id/ata-Samsung_SSD_850_EVO_500GB_S21LKWKGB18228E    

# format the disk into:
# 512M EFI, 2GB zfs boot pool, zfs root pool filling everything but the last 8GB, 8GB swap partition at the end of the disk
sudo sgdisk --zap-all $DISK
sudo sgdisk -n1:0:+512M -t1:EF00 $DISK
sudo sgdisk -n2:0:+2G $DISK   
sudo sgdisk -n3:0:-8G $DISK     
sudo sgdisk -n4:0:0 $DISK

# create random UUID to append to the zfs pool names so that importing the pools on other computers e.g. for rescue or backing them up won't result into name conflicts
poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)

# The boot-pool called bpool will be created with the following features enabled after checking
# https://wiki.archlinux.org/index.php/ZFS#GRUB-compatible_pool_creation
sudo zpool create \
    -o ashift=12 -o autotrim=on -d \
    -o feature@allocation_classes=enabled \
    -o feature@async_destroy=enabled \
    -o feature@bookmarks=enabled \
    -o feature@embedded_data=enabled \
    -o feature@empty_bpobj=enabled \
    -o feature@enabled_txg=enabled \
    -o feature@extensible_dataset=enabled \
    -o feature@filesystem_limits=enabled \
    -o feature@hole_birth=enabled \
    -o feature@large_blocks=enabled \
    -o feature@lz4_compress=enabled \
    -o feature@project_quota=enabled      \
    -o feature@resilver_defer=enabled     \
    -o feature@spacemap_histogram=enabled \
    -o feature@spacemap_v2=enabled        \
    -o feature@userobj_accounting=enabled \
    -o feature@zpool_checkpoint=enabled   \
    -O acltype=posixacl -O canmount=off -O compression=lz4 \
    -O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
    -O mountpoint=/boot -R /mnt \
    bpool_$poolUUID ${DISK}-part2

# create root-pool following settings list from:
# https://wiki.archlinux.org/index.php/Install_Arch_Linux_on_ZFS#Setup_the_ZFS_filesystem
# For root pool all available features are enabled by default
sudo zpool create \
	-f -o ashift=12 -o autotrim=on \
	-O acltype=posixacl       \
    -O relatime=on            \
    -O xattr=sa               \
    -O dnodesize=legacy       \
    -O normalization=formD    \
    -O mountpoint=none        \
    -O canmount=off           \
    -O devices=off            \
    -R /mnt                   \
    -O compression=lz4        \
    -O encryption=aes-256-gcm \
    -O keyformat=passphrase   \
    -O keylocation=prompt     \
    rpool_$poolUUID ${DISK}-part3
Password: password

# creating datasets. Nothing is stored directly under bpool and rpool, hence: canmount=off!
# canmount is set to noauto as mentioned here: https://wiki.archlinux.org/index.php/Install_Arch_Linux_on_ZFS#Create_your_datasets
sudo zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
sudo zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
sudo zfs create -o canmount=off -o mountpoint=none bpool_$poolUUID/BOOT
sudo zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
sudo zfs mount rpool_$poolUUID/ROOT/default
sudo zfs create -o mountpoint=legacy bpool_$poolUUID/BOOT/default
sudo mkdir /mnt/boot
sudo mount -t zfs bpool_$poolUUID/BOOT/default /mnt/boot
sudo zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
sudo zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root

# formatting and mounting the EFI partition
sudo mkfs.vfat -n EFI $DISK-part1
sudo mkdir /mnt/boot/efi
sudo mount $DISK-part1 /mnt/boot/efi

# configuring the root filesystem as advised here:
# https://wiki.archlinux.org/index.php/Install_Arch_Linux_on_ZFS#Configure_the_root_filesystem
sudo zpool set bootfs=rpool_$poolUUID/ROOT/default rpool_$poolUUID
sudo zpool set cachefile=/etc/zfs/zpool.cache rpool_$poolUUID
sudo zpool set cachefile=/etc/zfs/zpool.cache bpool_$poolUUID

# Now install the base Manjaro system using the "setup" command from the livecd
# don't forget to add additional packages to the installation needed are:
1. linux510-headers
2. linux54-headers
3. zfs-dmks
4. cryptsetup
# use the installer to setup system/configure base and generate the fstab
# chroot into installation 
# comment out all non-legacy datasets in the /etc/fstab apart from the boot/EFI partition
# and the bpool_$poolUUID/BOOT/default partition
mkinitcpio -P
exit

# copy over the zpool.cache file to the new systemsu
sudo cp /etc/zfs/zpool.cache /mnt/etc/zfs/zpool.cache

# add encrypted swap partition to the system
# swap $DISK-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256
# to /mnt/etc/crypttab 
# /dev/mapper/swap none swap defaults 0 0
# to /mnt/etc/fstab

# chroot into installation at /mnt to install grub
sudo pacman -S grub efibootmgr
ZPOOL_VDEV_NAME_PATH=1 grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=Manjaro

# now we need to fix some issues in regards to grub

# 1. first we need to apply a fix as mentioned here: https://wiki.archlinux.org/index.php/Install_Arch_Linux_on_ZFS#Root_pool_missing_from_grub.cfg
# to do this edit /mnt/etc/grub.d/10_linux and replace the line starting with "rpool=" with the following string
rpool=`${grub_probe} --device ${GRUB_DEVICE} --target=fs_label 
2>/dev/null || zdb -l ${GRUB_DEVICE} | awk -F \' '/ name/ { print $2 }'`

# 2. chroot into installation at /mnt and run
sudo mkinitcpio -P
ZPOOL_VDEV_NAME_PATH=1 grub-mkconfig -o /boot/grub/grub.cfg

# following the configure systemd ZFS mounts section at:
# https://wiki.archlinux.org/index.php/Install_Arch_Linux_on_ZFS#Configure_systemd_ZFS_mounts
sudo zpool set cachefile=/etc/zfs/zpool.cache rpool_$poolUUID
sudo zpool set cachefile=/etc/zfs/zpool.cache bpool_$poolUUID
sudo systemctl enable zfs.target --root=/mnt
sudo systemctl enable zfs-import-cache --root=/mnt
sudo systemctl enable zfs-mount --root=/mnt
sudo systemctl enable zfs-import.target --root=/mnt
manjaro-chroot /mnt zgenhostid 19841984 
manjaro-chroot /mnt mkinitcpio -P

sudo umount /mnt/boot/efi
sudo umount /mnt/boot
sudo zfs umount -a
sudo zfs umount rpool_$poolUUID/ROOT/default
sudo zpool export bpool_$poolUUID
sudo zpool export rpool_$poolUUID

The zfs support in manjaro-architect is still pretty rudimentary, I’m not sure if it supports native encryption at all. @dalto might know something about this. Looking at the commands, you seem to be doing most of the zfs stuff manually anyway though…

It seems that

  • root partition is mounted correctly
  • decryption sequence is triggered
  • something goes wrong right there and systemd trips over itself

I can only point you here for documentation unfortunately:

https://wiki.archlinux.org/index.php/ZFS#Native_encryption

I would check if you have the correct hooks in your mkinitcpio.conf

2 Likes

It doesn’t look like he is using the zfs support in architect for the most part.

That being said, the installer is probably detecting a zfs root and may be stepping on some of the things he is trying to do by hand since it also does some of those same things.

I have used Manjaro-Architect do install and encrpyted zfs setup. The only things I had to do were:

  • Create the pools and setup the encryption outside of architect
  • Mount the pools from within architect
  • Install using architect as normal
  • Install a cpio hook with zupport for zfs encryption(not sure if this is still needed. At the time the Manjaro zfs hook was out-of-date)

The difference is I used systemd-boot which has native support for zfs encryption and doesn’t require any special handling like grub seems to.

3 Likes

No I am using it only for the installation procedure for the packaged but setup the partitions on my own.

Maybe using the architect itself to mount the partitions might be a good step.

But then I guess you do not have a boot pool, am I correct? The arch wiki says:
" Systemd-boot can’t open ZFS zpools, you must store your /boot on a separated VFAT or ext4 partition." - Install Arch Linux on ZFS - ArchWiki

@dalto @Chrysostomus Do you guys know any good explanation how to setup Manjaro with systemd-boot? My experience with systemd-boot is none. I like to try it out, but as far as I understand I need to create a boot entry on my own and do not know what I need to write there.

systemd-boot generally stores the kernels in the EFI partition. If your EFI partition is mounted at /boot than it needs to be fat32. If you efi partition is mount at /efi than the kernels will need to be there as well but then /boot can be any filesystem you want.

When you get to the bootloader section in Architect, select systemd-boot

Also, if you use something other than grub you won’t need to have separate zpools for encrypted and unencrypted data.

You can create a single zpool and instead of encrypting the zpool you can encrypt the datasets in the pool. This gives you more flexibility if you want it. For example, you could something like this assuming your zpool is named “zp”.

zp/encrypt/ROOT/manjaro
zp/encrypt/HOME

If you encrypt the dataset “encrypt”, everything under it will be encrypted but you have the flexibility to manage unencrypted data in your zpool if you ever need to. Since many zpool elements can’t be modified after the fact, I recommend setting as few as possible settings on your zpool and instead setting everything you can on the datasets. Having to destroy an entire zpool because 1 setting needs to be changed gets annoying fast.

FWIW, I boot 4 different Linux distros out of a single zpool. All are encrypted and they all use systemd-boot.

Out of curiosity, is there any reason to set canmount=off when mountpoint=none?

1 Like

I see, thanks for the explanation!

Thanks for Grub you need to configure quite some stuff. If you say it works just like that I will try. Thanks to KVM I will give it another try at installing this evening.

Thanks for the tip, but I certainly don’t want unencrypted stuff on disk for the Manjaro systems.

Good hint thanks. Most of the features that I set during pool creation were do to the fact that either GRUB needed them set to even work on a zfs formatted pool or Linux needs them in general. But I will follow your example and set them for the datasets.

Yeah, reason was just good practice. Since canmount is inherited by the datasets, it becomes the standard for all datasets. So if you want something to be mounted automatically you specifically need to set it to on. Just in case you set a mountpoint somewhere you need to also tell the dataset that it can mount on its own.

One other thing, I probably wouldn’t use the dkms modules with zfs on root. Use the manjaro provided modules instead.

Otherwise you can end up with an unbootable system when you get a kernel update that isn’t compatible with the dkms modules.

2 Likes

Do you mean not install any additional packages at all or instead of zfs-dkms I should rather install linux510-zfs?

OK, so I have changed the installation process to accommodate the usage of systemd-boot.
During the usage of manjaro-architect, during the step where I install the the bootloader I get the message that everything went fine, but closing the manjaro-architect I get the message that no bootloader is installed. Trying to boot the installation does not work at all since I land in the UEFIs shell since it does not seem to find a bootloader.

@dalto : I would really appreciate it if you could find the time to look over of what I have done, if you can spot any mistakes or have any tips what could be wrong here.

I do the following steps now during installation:

# create a short handle to reference the ssd by id
DISK=/dev/disk/by-id/ata-Samsung_SSD_850_EVO_500GB_S21LKWKGB18228E    

# format the disk into:
# 512M EFI, 2GB zfs boot pool, zfs root pool filling everything but the last 8GB, 8GB swap partition at the end of the disk
sudo sgdisk --zap-all $DISK
sudo sgdisk -n1:0:+2G -t1:EF00 $DISK 
sudo sgdisk -n2:0:-8G $DISK     
sudo sgdisk -n3:0:0 $DISK

# create random UUID to append to the zfs pool names so that importing the pools on other computers e.g. for rescue or backing them up won't result into name conflicts
poolUUID=$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6)

# create root-pool following settings list from:
# https://wiki.archlinux.org/index.php/Install_Arch_Linux_on_ZFS#Setup_the_ZFS_filesystem
# For root pool all available features are enabled by default
sudo zpool create \
	-f -o ashift=12 -o autotrim=on \
	-O acltype=posixacl       \
    -O relatime=on            \
    -O xattr=sa               \
    -O dnodesize=legacy       \
    -O normalization=formD    \
    -O mountpoint=none        \
    -O canmount=off           \
    -O devices=off            \
    -R /mnt                   \
    -O compression=lz4        \
    -O encryption=aes-256-gcm \
    -O keyformat=passphrase   \
    -O keylocation=prompt     \
    rpool_$poolUUID ${DISK}-part2

# creating datasets. Nothing is stored directly under rpool, hence: canmount=off!
# canmount is set to noauto as mentioned here: https://wiki.archlinux.org/index.php/Install_Arch_Linux_on_ZFS#Create_your_datasets
sudo zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/HOME
sudo zfs create -o canmount=off -o mountpoint=none rpool_$poolUUID/ROOT
sudo zfs create -o mountpoint=/ -o canmount=noauto rpool_$poolUUID/ROOT/default
sudo zfs mount rpool_$poolUUID/ROOT/default   
sudo zfs create -o mountpoint=/home rpool_$poolUUID/HOME/default
sudo zfs create -o mountpoint=/root rpool_$poolUUID/HOME/default/root
sudo zfs mount -a

# formatting and mounting the EFI partition
sudo mkfs.vfat -n EFI $DISK-part1
sudo mkdir /mnt/boot
sudo mount $DISK-part1 /mnt/boot

# configuring the root filesystem as advised here:
# https://wiki.archlinux.org/index.php/Install_Arch_Linux_on_ZFS#Configure_the_root_filesystem
sudo zpool set bootfs=rpool_$poolUUID/ROOT/default rpool_$poolUUID
sudo zpool set cachefile=/etc/zfs/zpool.cache rpool_$poolUUID

# Setup
Prepare Installation -> Set Virtual Console -> change -> "de" 
Install Desktop System -> Install Manjaro Desktop -> yay + base-devel; linux510; linux54 -> gnome -> Yes -> cryptsetup; linux54-zfs; linux510-zfs -> minimal -> Auto-install free drivers
Install Desktop System -> Install Bootloader -> systemd-boot -> Yes
Install Desktop System -> Configure Base -> Generate FSTAB -> fstabgen -U -p
Install Desktop System -> Configure Base -> Set Hostname
Install Desktop System -> Configure Base -> Set System locale -> en_US.UTF-8 -> de_DE.UTF-8
Install Desktop System -> Configure Base -> Set Desktop Keybaord Layout -> "de"
Install Desktop System -> Configure Base -> Set Timezone and Clock -> Europe -> Berlin -> Yes -> utc
Install Desktop System -> Configure Base -> Set Root Password 
Install Desktop System -> Configure Base -> Add New User(s) -> "user" -> bash
Done -> Yes -> Yes

# leaving the setup unmounts the zfs datasets and the efi partition so we need to remount them
sudo zfs mount rpool_$poolUUID/ROOT/default   
sudo zfs mount -a
sudo mount $DISK-part1 /mnt/boot

# copy over the zpool.cache file to the new systemsu
sudo cp /etc/zfs/zpool.cache /mnt/etc/zfs/zpool.cache

# add encrypted swap partition to the system
# "swap $DISK-part3 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256"
# to /mnt/etc/crypttab 
# "/dev/mapper/swap none swap defaults 0 0"
# to /mnt/etc/fstab

# following the configure systemd ZFS mounts section at:
# https://wiki.archlinux.org/index.php/Install_Arch_Linux_on_ZFS#Configure_systemd_ZFS_mounts
sudo systemctl enable zfs.target --root=/mnt
sudo systemctl enable zfs-import-cache --root=/mnt
sudo systemctl enable zfs-mount --root=/mnt
sudo systemctl enable zfs-import.target --root=/mnt
manjaro-chroot /mnt zgenhostid 19841984 
manjaro-chroot /mnt mkinitcpio -P

sudo umount /mnt/boot
sudo zfs umount -a
sudo zfs umount rpool_$poolUUID/ROOT/default
sudo zpool export rpool_$poolUUID

This. The reason is there are times where a new kernel won’t have zfs support. If you use the dkms modules you can get a new kernel that breaks your access to the root filesystem.

  1. It looks like you are mounting your filesystems before running architect. Unless it has changed recently architect unmounts filesystems at some points in the process. I wonder if something isn’t mounted when it needs to be.
  2. Make sure you boot the ISO in uefi mode. If you boot the ISO in legacy mode the bootloader install will probably fail.
  3. It has been a while since I looked at the code but there are some variables that architect sets and then references later. Skipping the mount step may be problematic. I would start architect with your zpool exported and nothing mounted in /mnt. Go into the zfs menu choose manual and ask it to import your pool. Then go into the mount menu and mount the stuff that needs to be mounted that way. As soon as you enter the moutn menu it should detect your zfs root.
  4. Not part of the problem but if you choose the option to get a shell from the installer before exiting it will drop you into a fully built chroot before unmounting so you won’t have to remount everything and run all those commands against /mnt or use manjaro-chroot.

The installer should be doing all this for you. It shouldn’t be needed to do it manually.

2 Likes

The Problem why I mounted everything by hand was that the Archwiki says that the root dataset must use canmount=noauto, however everything that is not mounted by zfs mount -a is not loaded by the installer upon importing the pool. There is also no option in the installer to manually mount a dataset.
Also the efi partition is not mounted and with the mount menu point the installer ask me to choose a partition to mount root from, I had not really a chance to choose to mount boot!

It is booted in EFI mode.

Ok I will try without it.

It definitely is not true that it is a hard requirement because I have over a dozen zfs on root installs and none of them do that. It is possible it is a best practice.

If you want that, why not just set it to canmount=noauto after the install is completed.

The installer handles the mounting of the efi partition specially. You don’t mount it. It asks you to choose the efi partition and after you select it, it asks you if you want to mount it on /boot or /boot/efi

1 Like

Are you sure? I tried starting the install without mounting the efi and get the following:

Also if I try to use the option to mount partitions I get stuck at:

At what point in the install are you seeing that?

That means that it didn’t detect your zfs root. Make sure that canmount=on on your / dataset. You can change it to noauto after the install is complete.

1 Like

Install Desktop System -> Install Manjaro Desktop

I already changed that during creation of the datasets. I then exported the pool and used the architects ZFS menu to mount the pool. However there is no option in the installer to check if everything was mounted correctly.

It gets mounted before that, at the end of the mount step. Maybe because your root is not mounted, you aren’t getting the prompts for that.

You can open a second terminal and check to see if everything was mounted.

1 Like

The ZFS options in the installer don’t work at all, at least not with natively encrypted pools. I can not even create new datasets with the architect on the encrypted pool neither mount any of them.

I have never tested them with encrypted pools so it may not work.

In that case try this.

  • Launch the architect installer
  • Import your pool with -R /mnt in another terminal
  • Go into the mount menu in architect
1 Like

Launched the architect installer, then imported the pool with -R /mnt in another terminal.

Then:
Prepare Installation -> ZFS (optional) -> Manual configuration -> Create and mount a ZFS filesystem -> choose “rpool_iw9sab” -> set dataset name and mountpoint

But after that when I go to:
Prepare Installation -> ZFS (optional) -> Manual configuration -> Destroy a ZFS dataset

The installer finds nothing. So I am pretty sure it did not create the dataset in the first place. It seems the installer can’t handle encrypted pools.

Edit: Instead of fiddeling the the buggy installer I would rather know how to install systemd-boot for Manjaro on my own, because then I can maybe do a manual install and set everything up myself.

You should skip this step if you manually mount it. Go straight to the “mount” menu.

There is nothing special about setting systemd-boot up on manjaro. You can either do it manually or use systemd-boot-manager.

If you want to use systemd-boot-manager you can simply:

sudo pacman -S systemd-boot-manager
sdboot-manage setup
1 Like