Failed boot after every update (ZFS on root)

Support
1

NOTE: This is probably exclusively related to the installation being ZFS-on-root

After each update on Manjaro (KDE) with ZFS-on-Root, the system fails to boot and ends up in emergency mode. I can always fix this by chrooting into my system from a USB Drive by doing the following:

  1. reinstall kernel & zfs module
    • pacman -Sy linux515 linux515-zfs
  2. regenerate the initramfs (does not work without step 1)
    • mkinitcpio -P
  3. regenerate grub config
    • ZPOOL_VDEV_NAME_PATH=1 grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=Manjaro
    • ZPOOL_VDEV_NAME_PATH=1 grub-mkconfig -o /boot/grub/grub.cfg

But it’s annoying to have to do this after each update. I’ve probably misconfigured something, so I’m posting this to ask if anyone has an idea on what might be a more permanent fix than having to go through all these steps after every update.

Here are the logs from the update last night (which caused the system to boot into emergency mode):

There was an error during the update which is not displayed here, something about “GRUB UPDATE FAILED”

[2022-08-09T18:16:30+0200] [PAMAC] synchronizing package lists
[2022-08-09T18:16:35+0200] [ALPM] running '00-timeshift-autosnap.hook'...
[2022-08-09T18:16:35+0200] [ALPM-SCRIPTLET] ==> skipping timeshift-autosnap due skipRsyncAutosnap in /etc/timeshift-autosnap.conf set to TRUE.
[2022-08-09T18:16:35+0200] [ALPM] transaction started
[2022-08-09T18:16:35+0200] [ALPM] upgraded archlinux-keyring (20220713-2 -> 20220727-1)
[2022-08-09T18:16:35+0200] [ALPM-SCRIPTLET] ==> Appending keys from archlinux.gpg...
[2022-08-09T18:16:38+0200] [ALPM-SCRIPTLET] gpg: public key DB323392796CA067 is 3037 days newer than the signature
[2022-08-09T18:16:39+0200] [ALPM-SCRIPTLET] ==> Updating trust database...
[2022-08-09T18:16:39+0200] [ALPM-SCRIPTLET] gpg: next trustdb check due at 2022-08-09
[2022-08-09T18:16:39+0200] [ALPM-SCRIPTLET] ==> Updating trust database...
[2022-08-09T18:16:39+0200] [ALPM-SCRIPTLET] gpg: next trustdb check due at 2022-08-09
[2022-08-09T18:16:39+0200] [ALPM] transaction completed
[2022-08-09T18:16:39+0200] [ALPM] running '30-systemd-update.hook'...
[2022-08-09T18:16:39+0200] [ALPM] running '90-packagekit-refresh.hook'...
[2022-08-09T18:18:58+0200] [ALPM] running '00-timeshift-autosnap.hook'...
[2022-08-09T18:18:58+0200] [ALPM-SCRIPTLET] ==> skipping timeshift-autosnap due skipRsyncAutosnap in /etc/timeshift-autosnap.conf set to TRUE.
[2022-08-09T18:18:58+0200] [ALPM] running '60-mkinitcpio-remove.hook'...
[2022-08-09T18:18:58+0200] [ALPM] running 'firefox-pre.hook'...
[2022-08-09T18:18:58+0200] [ALPM] running 'thunderbird-pre.hook'...
[2022-08-09T18:18:58+0200] [ALPM] transaction started
[2022-08-09T18:18:58+0200] [ALPM] upgraded linux-api-headers (5.17.5-2 -> 5.18.15-1)
[2022-08-09T18:18:58+0200] [ALPM] warning: /etc/locale.gen installed as /etc/locale.gen.pacnew
[2022-08-09T18:18:58+0200] [ALPM] upgraded glibc (2.35-6 -> 2.36-1)
[2022-08-09T18:18:58+0200] [ALPM-SCRIPTLET] Generating locales...
[2022-08-09T18:18:59+0200] [ALPM-SCRIPTLET]   en_US.UTF-8... done
[2022-08-09T18:18:59+0200] [ALPM-SCRIPTLET] Generation complete.
[2022-08-09T18:19:00+0200] [ALPM] upgraded gcc-libs (12.1.0-3 -> 12.1.1-4)
[2022-08-09T18:19:00+0200] [ALPM] upgraded lib32-glibc (2.35-6 -> 2.36-1)
[2022-08-09T18:19:00+0200] [ALPM] upgraded libelf (0.187-1 -> 0.187-2)
[2022-08-09T18:19:00+0200] [ALPM] upgraded binutils (2.38-6 -> 2.38-7)
[2022-08-09T18:19:00+0200] [ALPM] upgraded libsysprof-capture (3.44.0-1 -> 3.44.0-2)
[2022-08-09T18:19:00+0200] [ALPM] upgraded harfbuzz (5.0.1-1 -> 5.1.0-1)
[2022-08-09T18:19:00+0200] [ALPM] upgraded gsm (1.0.20-1 -> 1.0.21-1)
[2022-08-09T18:19:00+0200] [ALPM] upgraded opencore-amr (0.1.5-5 -> 0.1.6-1)
[2022-08-09T18:19:00+0200] [ALPM] upgraded cryptsetup (2.4.3-2 -> 2.5.0-1)
[2022-08-09T18:19:00+0200] [ALPM] upgraded iptables (1:1.8.8-1 -> 1:1.8.8-2)
[2022-08-09T18:19:00+0200] [ALPM] upgraded llvm-libs (14.0.6-2 -> 14.0.6-3)
[2022-08-09T18:19:00+0200] [ALPM] upgraded libtiff (4.4.0-1 -> 4.4.0-3)
[2022-08-09T18:19:01+0200] [ALPM] upgraded chromium (103.0.5060.134-1 -> 104.0.5112.79-1)
[2022-08-09T18:19:01+0200] [ALPM] upgraded mpfr (4.1.0.p13-2 -> 4.1.0.p13-3)
[2022-08-09T18:19:02+0200] [ALPM] upgraded gcc (12.1.0-3 -> 12.1.1-4)
[2022-08-09T18:19:02+0200] [ALPM] upgraded clang (14.0.6-1 -> 14.0.6-2)
[2022-08-09T18:19:02+0200] [ALPM] upgraded libarchive (3.6.1-1 -> 3.6.1-2)
[2022-08-09T18:19:03+0200] [ALPM] upgraded cmake (3.23.2-2 -> 3.23.3-1)
[2022-08-09T18:19:04+0200] [ALPM] upgraded firefox (103.0-1 -> 103.0.1-1)
[2022-08-09T18:19:04+0200] [ALPM] upgraded game-devices-udev (0.16-1 -> 0.17-1)
[2022-08-09T18:19:04+0200] [ALPM] upgraded vim-runtime (9.0.0000-1 -> 9.0.0135-1)
[2022-08-09T18:19:04+0200] [ALPM] upgraded libtool (2.4.7-2 -> 2.4.7-3)
[2022-08-09T18:19:04+0200] [ALPM] upgraded gvim (9.0.0000-1 -> 9.0.0135-1)
[2022-08-09T18:19:04+0200] [ALPM] upgraded harfbuzz-icu (5.0.1-1 -> 5.1.0-1)
[2022-08-09T18:19:04+0200] [ALPM] upgraded imagemagick (7.1.0.44-1 -> 7.1.0.45-1)
[2022-08-09T18:19:04+0200] [ALPM] upgraded iproute2 (5.18.0-1 -> 5.19.0-1)
[2022-08-09T18:19:04+0200] [ALPM] upgraded tevent (1:0.12.1-1 -> 1:0.13.0-1)
[2022-08-09T18:19:04+0200] [ALPM] upgraded ldb (2:2.5.2-1 -> 2:2.5.2-2)
[2022-08-09T18:19:04+0200] [ALPM] upgraded lib32-gcc-libs (12.1.0-3 -> 12.1.1-4)
[2022-08-09T18:19:04+0200] [ALPM] upgraded lib32-libxcrypt (4.4.28-1 -> 4.4.28-2)
[2022-08-09T18:19:04+0200] [ALPM] upgraded lib32-harfbuzz (5.0.1-1 -> 5.1.0-1)
[2022-08-09T18:19:04+0200] [ALPM] upgraded lib32-libtiff (4.4.0-1 -> 4.4.0-3)
[2022-08-09T18:19:04+0200] [ALPM] upgraded lib32-wayland (1.20.0-2 -> 1.21.0-1)
[2022-08-09T18:19:04+0200] [ALPM] upgraded lib32-libva (2.14.0-1 -> 2.15.0-1)
[2022-08-09T18:19:04+0200] [ALPM] upgraded mpg123 (1.30.1-1 -> 1.30.2-1)
[2022-08-09T18:19:04+0200] [ALPM] upgraded lib32-mpg123 (1.30.1-1 -> 1.30.2-1)
[2022-08-09T18:19:04+0200] [ALPM] upgraded sdl2_image (2.6.0-2 -> 2.6.1-1)
[2022-08-09T18:19:04+0200] [ALPM] upgraded lib32-sdl2_image (2.0.5-2 -> 2.6.0-1)
[2022-08-09T18:19:04+0200] [ALPM] upgraded lib32-sdl2_mixer (2.0.4-2 -> 2.6.1-1)
[2022-08-09T18:19:04+0200] [ALPM] upgraded lib32-sdl2_ttf (2.0.18-1 -> 2.20.0-1)
[2022-08-09T18:19:05+0200] [ALPM] upgraded python (3.10.5-1 -> 3.10.5-2)
[2022-08-09T18:19:07+0200] [ALPM] upgraded libreoffice-fresh (7.3.4-2 -> 7.3.5-1)
[2022-08-09T18:19:07+0200] [ALPM] upgraded libupnp (1.14.12-3 -> 1.14.13-1)
[2022-08-09T18:19:07+0200] [ALPM] upgraded libwpe (1.12.0-2 -> 1.12.2-1)
[2022-08-09T18:19:08+0200] [ALPM] upgraded linux515 (5.15.57-2 -> 5.15.59-1)
[2022-08-09T18:19:08+0200] [ALPM] upgraded linux515-zfs (2.1.5-10 -> 2.1.5-13)
[2022-08-09T18:19:18+0200] [ALPM] upgraded llvm (14.0.6-2 -> 14.0.6-3)
[2022-08-09T18:19:18+0200] [ALPM] upgraded mhwd-nvidia (515.57-1 -> 515.65.01-2)
[2022-08-09T18:19:18+0200] [ALPM] upgraded mhwd-nvidia-390xx (390.151-2 -> 390.154-1)
[2022-08-09T18:19:18+0200] [ALPM] upgraded mhwd-nvidia-470xx (470.129.06-1 -> 470.141.03-1)
[2022-08-09T18:19:18+0200] [ALPM] upgraded networkmanager-l2tp (1.20.0-3 -> 1.20.4-1)
[2022-08-09T18:19:18+0200] [ALPM] upgraded npm (8.15.1-1 -> 8.16.0-1)
[2022-08-09T18:19:18+0200] [ALPM] upgraded perl-alien-build (2.50-1 -> 2.51-1)
[2022-08-09T18:19:18+0200] [ALPM] upgraded perl-libwww (6.60-2 -> 6.67-1)
[2022-08-09T18:19:18+0200] [ALPM] upgraded python-chardet (4.0.0-5 -> 5.0.0-1)
[2022-08-09T18:19:18+0200] [ALPM] upgraded python-reportlab (3.6.10-1 -> 3.6.11-1)
[2022-08-09T18:19:18+0200] [ALPM] upgraded python-requests (2.27.1-1 -> 2.28.1-1)
[2022-08-09T18:19:19+0200] [ALPM] upgraded qt5-base (5.15.5+kde+r173-1 -> 5.15.5+kde+r174-1)
[2022-08-09T18:19:19+0200] [ALPM] upgraded smbclient (4.16.4-1 -> 4.16.4-2)
[2022-08-09T18:19:19+0200] [ALPM] upgraded samba (4.16.4-1 -> 4.16.4-2)
[2022-08-09T18:19:20+0200] [ALPM] upgraded signal-desktop (5.51.1-1 -> 5.52.0-1)
[2022-08-09T18:19:20+0200] [ALPM] installed libxcrypt-compat (4.4.28-2)
[2022-08-09T18:19:20+0200] [ALPM] installed lib32-libxcrypt-compat (4.4.28-4)
[2022-08-09T18:19:21+0200] [ALPM] upgraded webkit2gtk-4.1 (2.36.4-2 -> 2.36.5-2)
[2022-08-09T18:19:21+0200] [ALPM] upgraded steam-manjaro (1.0.0.74-1 -> 1.0.0.75-1)
[2022-08-09T18:19:21+0200] [ALPM] upgraded thunderbird (91.11.0-1 -> 91.12.0-1)
[2022-08-09T18:19:21+0200] [ALPM] upgraded wavpack (5.4.0-1 -> 5.5.0-1)
[2022-08-09T18:19:21+0200] [ALPM] upgraded wayland-protocols (1.25-1 -> 1.26-1)
[2022-08-09T18:19:21+0200] [ALPM] upgraded webkit2gtk (2.36.4-2 -> 2.36.5-2)
[2022-08-09T18:19:23+0200] [ALPM] upgraded wine (7.13-1 -> 7.14-1)
[2022-08-09T18:19:23+0200] [ALPM] upgraded xdg-desktop-portal (1.14.5-1 -> 1.14.6-1)
[2022-08-09T18:19:23+0200] [ALPM] transaction completed
[2022-08-09T18:19:23+0200] [ALPM] running '30-systemd-binfmt.hook'...
[2022-08-09T18:19:23+0200] [ALPM] running '30-systemd-daemon-reload.hook'...
[2022-08-09T18:19:23+0200] [ALPM] running '30-systemd-tmpfiles.hook'...
[2022-08-09T18:19:23+0200] [ALPM] running '30-systemd-udev-reload.hook'...
[2022-08-09T18:19:23+0200] [ALPM] running '30-systemd-update.hook'...
[2022-08-09T18:19:23+0200] [ALPM] running '30-update-mime-database.hook'...
[2022-08-09T18:19:24+0200] [ALPM] running '60-depmod.hook'...
[2022-08-09T18:19:32+0200] [ALPM] running '60-glibc-locales.hook'...
[2022-08-09T18:19:33+0200] [ALPM] running '80-cronie.hook'...
[2022-08-09T18:19:33+0200] [ALPM] running '90-mkinitcpio-install.hook'...
[2022-08-09T18:19:33+0200] [ALPM-SCRIPTLET] ==> Building image from preset: /etc/mkinitcpio.d/linux515.preset: 'default'
[2022-08-09T18:19:33+0200] [ALPM-SCRIPTLET]   -> -k /boot/vmlinuz-5.15-x86_64 -c /etc/mkinitcpio.conf -g /boot/initramfs-5.15-x86_64.img
[2022-08-09T18:19:33+0200] [ALPM-SCRIPTLET] ==> Starting build: 5.15.59-1-MANJARO
[2022-08-09T18:19:33+0200] [ALPM-SCRIPTLET]   -> Running build hook: [base]
[2022-08-09T18:19:33+0200] [ALPM-SCRIPTLET]   -> Running build hook: [udev]
[2022-08-09T18:19:34+0200] [ALPM-SCRIPTLET]   -> Running build hook: [autodetect]
[2022-08-09T18:19:34+0200] [ALPM-SCRIPTLET]   -> Running build hook: [modconf]
[2022-08-09T18:19:34+0200] [ALPM-SCRIPTLET]   -> Running build hook: [block]
[2022-08-09T18:19:34+0200] [ALPM-SCRIPTLET]   -> Running build hook: [keyboard]
[2022-08-09T18:19:34+0200] [ALPM-SCRIPTLET]   -> Running build hook: [zfs]
[2022-08-09T18:19:36+0200] [ALPM-SCRIPTLET]   -> Running build hook: [filesystems]
[2022-08-09T18:19:37+0200] [ALPM-SCRIPTLET] ==> Generating module dependencies
[2022-08-09T18:19:37+0200] [ALPM-SCRIPTLET] ==> Creating gzip-compressed initcpio image: /boot/initramfs-5.15-x86_64.img
[2022-08-09T18:19:43+0200] [ALPM-SCRIPTLET] ==> Image generation successful
[2022-08-09T18:19:43+0200] [ALPM-SCRIPTLET] ==> Building image from preset: /etc/mkinitcpio.d/linux515.preset: 'fallback'
[2022-08-09T18:19:43+0200] [ALPM-SCRIPTLET]   -> -k /boot/vmlinuz-5.15-x86_64 -c /etc/mkinitcpio.conf -g /boot/initramfs-5.15-x86_64-fallback.img -S autodetect
[2022-08-09T18:19:43+0200] [ALPM-SCRIPTLET] ==> Starting build: 5.15.59-1-MANJARO
[2022-08-09T18:19:43+0200] [ALPM-SCRIPTLET]   -> Running build hook: [base]
[2022-08-09T18:19:43+0200] [ALPM-SCRIPTLET]   -> Running build hook: [udev]
[2022-08-09T18:19:44+0200] [ALPM-SCRIPTLET]   -> Running build hook: [modconf]
[2022-08-09T18:19:44+0200] [ALPM-SCRIPTLET]   -> Running build hook: [block]
[2022-08-09T18:19:44+0200] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: bfa
[2022-08-09T18:19:45+0200] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: qed
[2022-08-09T18:19:45+0200] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: qla1280
[2022-08-09T18:19:45+0200] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: qla2xxx
[2022-08-09T18:19:47+0200] [ALPM-SCRIPTLET]   -> Running build hook: [keyboard]
[2022-08-09T18:19:48+0200] [ALPM-SCRIPTLET]   -> Running build hook: [zfs]
[2022-08-09T18:19:50+0200] [ALPM-SCRIPTLET]   -> Running build hook: [filesystems]
[2022-08-09T18:19:54+0200] [ALPM-SCRIPTLET] ==> Generating module dependencies
[2022-08-09T18:19:54+0200] [ALPM-SCRIPTLET] ==> Creating gzip-compressed initcpio image: /boot/initramfs-5.15-x86_64-fallback.img
[2022-08-09T18:20:03+0200] [ALPM-SCRIPTLET] ==> Image generation successful
[2022-08-09T18:20:03+0200] [ALPM] running '90-packagekit-refresh.hook'...
[2022-08-09T18:20:03+0200] [ALPM] running '99-grub.hook'...
[2022-08-09T18:20:03+0200] [ALPM-SCRIPTLET] /usr/bin/grub-probe: error: failed to get canonical path of `/dev/nvme-WD_BLACK_SN750_SE_1TB_21440T800632-part2'.
[2022-08-09T18:20:03+0200] [ALPM] running 'dbus-reload.hook'...
[2022-08-09T18:20:03+0200] [ALPM] running 'detect-old-perl-modules.hook'...
[2022-08-09T18:20:03+0200] [ALPM] running 'firefox-post.hook'...
[2022-08-09T18:20:03+0200] [ALPM] running 'fontconfig.hook'...
[2022-08-09T18:20:04+0200] [ALPM] running 'fontconfig-32.hook'...
[2022-08-09T18:20:04+0200] [ALPM] running 'gtk-update-icon-cache.hook'...
[2022-08-09T18:20:04+0200] [ALPM] running 'locale-gen.hook'...
[2022-08-09T18:20:04+0200] [ALPM] running 'texinfo-install.hook'...
[2022-08-09T18:20:05+0200] [ALPM] running 'thunderbird-post.hook'...
[2022-08-09T18:20:05+0200] [ALPM] running 'update-desktop-database.hook'...

Here are logs of the fix I mentioned earlier (while chrooted), which fixes the problem (successful boot back to KDE):

[2022-08-09T18:24:09+0200] [PACMAN] Running 'pacman -Sy linux-515 linux-515-zfs'
[2022-08-09T18:24:09+0200] [PACMAN] synchronizing package lists
[2022-08-09T18:24:18+0200] [PACMAN] Running 'pacman -Sy linux515 linux515-zfs'
[2022-08-09T18:24:18+0200] [PACMAN] synchronizing package lists
[2022-08-09T18:24:21+0200] [ALPM] running '00-timeshift-autosnap.hook'...
[2022-08-09T18:24:21+0200] [ALPM-SCRIPTLET] ==> skipping timeshift-autosnap due skipRsyncAutosnap in /etc/timeshift-autosnap.conf set to TRUE.
[2022-08-09T18:24:21+0200] [ALPM] transaction started
[2022-08-09T18:24:22+0200] [ALPM] reinstalled linux515 (5.15.59-1)
[2022-08-09T18:24:22+0200] [ALPM] reinstalled linux515-zfs (2.1.5-13)
[2022-08-09T18:24:31+0200] [ALPM] transaction completed
[2022-08-09T18:24:31+0200] [ALPM] running '30-systemd-update.hook'...
[2022-08-09T18:24:31+0200] [ALPM] running '60-depmod.hook'...
[2022-08-09T18:24:39+0200] [ALPM] running '90-mkinitcpio-install.hook'...
[2022-08-09T18:24:39+0200] [ALPM-SCRIPTLET] ==> Building image from preset: /etc/mkinitcpio.d/linux515.preset: 'default'
[2022-08-09T18:24:39+0200] [ALPM-SCRIPTLET]   -> -k /boot/vmlinuz-5.15-x86_64 -c /etc/mkinitcpio.conf -g /boot/initramfs-5.15-x86_64.img
[2022-08-09T18:24:39+0200] [ALPM-SCRIPTLET] ==> Starting build: 5.15.59-1-MANJARO
[2022-08-09T18:24:39+0200] [ALPM-SCRIPTLET]   -> Running build hook: [base]
[2022-08-09T18:24:39+0200] [ALPM-SCRIPTLET]   -> Running build hook: [udev]
[2022-08-09T18:24:40+0200] [ALPM-SCRIPTLET]   -> Running build hook: [autodetect]
[2022-08-09T18:24:40+0200] [ALPM-SCRIPTLET]   -> Running build hook: [modconf]
[2022-08-09T18:24:40+0200] [ALPM-SCRIPTLET]   -> Running build hook: [block]
[2022-08-09T18:24:40+0200] [ALPM-SCRIPTLET]   -> Running build hook: [keyboard]
[2022-08-09T18:24:40+0200] [ALPM-SCRIPTLET]   -> Running build hook: [zfs]
[2022-08-09T18:24:42+0200] [ALPM-SCRIPTLET]   -> Running build hook: [filesystems]
[2022-08-09T18:24:43+0200] [ALPM-SCRIPTLET] ==> Generating module dependencies
[2022-08-09T18:24:43+0200] [ALPM-SCRIPTLET] ==> Creating gzip-compressed initcpio image: /boot/initramfs-5.15-x86_64.img
[2022-08-09T18:24:49+0200] [ALPM-SCRIPTLET] ==> Image generation successful
[2022-08-09T18:24:49+0200] [ALPM-SCRIPTLET] ==> Building image from preset: /etc/mkinitcpio.d/linux515.preset: 'fallback'
[2022-08-09T18:24:49+0200] [ALPM-SCRIPTLET]   -> -k /boot/vmlinuz-5.15-x86_64 -c /etc/mkinitcpio.conf -g /boot/initramfs-5.15-x86_64-fallback.img -S autodetect
[2022-08-09T18:24:49+0200] [ALPM-SCRIPTLET] ==> Starting build: 5.15.59-1-MANJARO
[2022-08-09T18:24:49+0200] [ALPM-SCRIPTLET]   -> Running build hook: [base]
[2022-08-09T18:24:49+0200] [ALPM-SCRIPTLET]   -> Running build hook: [udev]
[2022-08-09T18:24:50+0200] [ALPM-SCRIPTLET]   -> Running build hook: [modconf]
[2022-08-09T18:24:50+0200] [ALPM-SCRIPTLET]   -> Running build hook: [block]
[2022-08-09T18:24:50+0200] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: bfa
[2022-08-09T18:24:51+0200] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: qed
[2022-08-09T18:24:51+0200] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: qla1280
[2022-08-09T18:24:51+0200] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: qla2xxx
[2022-08-09T18:24:53+0200] [ALPM-SCRIPTLET]   -> Running build hook: [keyboard]
[2022-08-09T18:24:54+0200] [ALPM-SCRIPTLET]   -> Running build hook: [zfs]
[2022-08-09T18:24:55+0200] [ALPM-SCRIPTLET]   -> Running build hook: [filesystems]
[2022-08-09T18:24:59+0200] [ALPM-SCRIPTLET] ==> Generating module dependencies
[2022-08-09T18:24:59+0200] [ALPM-SCRIPTLET] ==> Creating gzip-compressed initcpio image: /boot/initramfs-5.15-x86_64-fallback.img
[2022-08-09T18:24:59+0200] [ALPM-SCRIPTLET] ==> Creating gzip-compressed initcpio image: /boot/initramfs-5.15-x86_64-fallback.img
[2022-08-09T18:25:09+0200] [ALPM-SCRIPTLET] ==> Image generation successful
[2022-08-09T18:25:09+0200] [ALPM] running '90-packagekit-refresh.hook'...
[2022-08-09T18:25:09+0200] [ALPM-SCRIPTLET] Error connecting: Could not connect: No such file or directory
[2022-08-09T18:25:09+0200] [ALPM] running '99-grub.hook'...
[2022-08-09T18:25:09+0200] [ALPM-SCRIPTLET] /usr/bin/grub-probe: error: failed to get canonical path of `/dev/nvme-WD_BLACK_SN750_SE_1TB_21440T800632-part2'.

Here are the journalctl logs for the system booting into emergency mode:

  • Aug 09 18:21:43 hummingbird systemd[1]: Reached target Emergency Mode.
Aug 09 18:21:42 hummingbird systemd-modules-load[825]: Failed to find module 'crypto_user'
Aug 09 18:21:42 hummingbird systemd-modules-load[825]: Failed to find module 'i2c-dev'
Aug 09 18:21:42 hummingbird systemd-modules-load[825]: Failed to find module 'uinput'
Aug 09 18:21:42 hummingbird systemd-modules-load[825]: Failed to find module 'zfs'
Aug 09 18:21:43 hummingbird systemd[1]: Failed to mount /boot.
Aug 09 18:21:43 hummingbird systemd[1]: Failed to mount /mnt/mediasnek.
Aug 09 18:21:49 hummingbird systemd[1]: Failed to mount /mnt/windows.
2

Maybe you can switch Grub to Systemd-boot, because Grub does not support ZFS on root well from what I’ve heard.

The other possibility is safe, that ZFS does not run on root at booting. You can use any official filesystem e.g. ext4 or xfs or btrfs on / , all the rest e.g. /home, /etc/ and /opt …, is on ZFS.

3

Maybe you can switch Grub to Systemd-boot

I certainly could, but I’d do that as a last resort. We have two Manjaro installs at home - both of them using grub as a bootloader.

  • The first is an XFCE install which was created with manjaro-architect (back in ~2019ish?) which has no issues with updates via pamac.
  • The second is a KDE install which was recently created following this guide which has the issue described in the OP.

This means it’s possible to have use grub with ZFS without having to do all the steps I mentioned.

The other possibility is safe, that ZFS does not run on root at booting. You can use any official filesystem e.g. ext4 or xfs or btrfs on / , all the rest e.g. /home, /etc/ and /opt …, is on ZFS.

I get what you’re saying, but I don’t see why it runs completely fine, with grub, but falls to pieces after an update and requires a manual fix. Surely there’s a way that doesn’t require changing bootloader, or ZFS datasets especially since it’s already working fine on one installation.

4

I told you that Grub does not support ZFS, for example, you change compression in any dataset that causes crashing Grub, I experienced this in the VM (my testing). I heard that systemd-boot or its other alternative has no issue with ZFS at booting.

I personally use ZFS mirror (RAID 1) in two other disks for important data/archives, backup and sharing like NAS. Not in root system. (Reason: your data is worth more than root system and apps)

5

Just would like to check if you only enable grub-compatible parameters for your zfs root pool?
Systemd-boot works well for me, so if grub doesn’t work, maybe try to switch to systemd-boot.

6

If you want to install full ZFS on root system without complicated setup, I know there is CachyOS that is the other arch based distro and supports systemd-boot and full ZFS on root system via calamares installer

7

I believe this is the issue. You probably need to add the environmental variable ZPOOL_VDEV_NAME_PATH=1 in the Pacman hook to make grub-probe use the correct path to identify your zfs pool.

8

Don’t use -Sy, that’s how you get a partial update, use -S or -Syu. Probably doesn’t matter too much if you’ve just updated.

Should be /dev/disk/by-id/nvme-WD_BLACK_SN750_SE_1TB_21440T800632-part2.

What ZFS features are you using for root? Surely it would be easier if you used EXT4…or BTRFS if you must have a COW. :grin:

@dobedobedo

Perhaps, but it happens with both the failed update and the fix. :man_shrugging:

9

I think you’re right! The missing environment variable is what is probably causing this issue.

Since there was a new update, I’ve made the following change to /usr/share/libalpm/hooks/99-grub.hook:

[Trigger]
Type = Path
Operation = Install
Operation = Upgrade
Operation = Remove
Target = usr/lib/modules/*/vmlinuz
Target = boot/vmlinuz*

[Action]
Description = Updating Grub-Bootmenu
When = PostTransaction
Exec = "ZPOOL_VDEV_NAME_PATH=1 /usr/bin/update-grub"

But it doesn’t work:
2022-08-14_11-15

I’m thinking of making it run sh -c "ZPOOL_VDEV_NAME_PATH=1 /usr/bin/update-grub" or execute a script file with this command, let me know, if you have better ideas - I’m not sure about the correct format for Exec = based on the manpage

What ZFS features are you using for root? Surely it would be easier if you used EXT4…or BTRFS if you must have a COW. :grin:

I was thinking of using BTRFS, but opted for ZFS since it’s what I’m already familiar with (we have a NAS with ZFS pools).

I wanted a COW filesystem on the root partition to be able to easily roll back in case something broke after an update.

10

You mentioned you have another system where the grub menu was updated normally. Maybe you should check what’s the differences between those pacman hook scripts.

11

Hi,
I’m just thinking that what if you just add the ZPOOL_VDEV_NAME_PATH=1 variable to /etc/environment?

12

There is a risk, if you set compression=zstd for any dataset. I am testing it in VM

$ sudo ZPOOL_VDEV_NAME_PATH=1 grub-mkconfig -o /boot/grub/grub.cfg

Generating grub configuration file ...
Found linux image: /boot/vmlinuz-linux
Found initrd image: /boot/amd-ucode.img /boot/initramfs-linux.img
/usr/bin/grub-probe: error: unknown filesystem. # --->  ISSUE: Unknown filesystem.
Found fallback initrd image(s) in /boot:  amd-ucode.img initramfs-linux-fallback.img
Warning: os-prober will be executed to detect other bootable partitions.
Its output will be used to detect bootable binaries on them and create new boot entries.
Adding boot menu entry for UEFI Firmware Settings ...
done

ZPOOL_VDEV_NAME_PATH=1 does not help.
You have to set back compression=off for Grub.

Systemd-boot has no issue with compression=zstd

13

Hi,
I’m just thinking that what if you just add the ZPOOL_VDEV_NAME_PATH=1 variable to /etc/environment?

@dobedobedo Thanks! I feel stupid for not thinking of that earlier, I’ll try it out next update.

@Zesko You might have found the problem, that is a difference between the zpool features of the two systems:

  • KDE (grub problems) feature@zstd_compress enabled local
  • XFCE (no grub issues) feature@zstd_compress disabled local
14

I believe you don’t need to wait until the next update. You can check

And also the content of the /etc/environment in the two systems.