21.2.2 Breaks systems running with a ZFS root

Support
,
1

WARNING for ZFS users –
Upgraded from 21.2 to latest 21.2.2 and it broke my system. Thankfully, had taken a snapshot before the upgrade, but still spent about 5 hours trying to figure stuff out, then finally admitting defeat and doing a rollback, and fixing things manually.

I think I narrowed down the problem being related to GRUB, and maybe something to do with zpool flags, but I’m not a developer, and I couldn’t give this any more time since it was my primary system that had failed.

Prior kernel - 5.15.12-1-MANJARO
New Kernel - 5.15.16-1-MANJARO

/boot is on BTRFS
/boot/EFI is unchanged (UEFI booting)

I kept dropping into rootfs on booting up, with the root zpool not mounting, something about a canonicalization error (WTF is that?), and even when I would mount everything manually and properly on /new_root, I’d get a kernel panic.

Tried loading 21.2.2 with 5.15.12-1-MANJARO kernel and initrd, that didn’t work either.

Tried re-installing all the kernel modules for the newer kernel, didn’t work.

“grub-probe /” didn’t work on 21.2, it doesn’t work on 21.2.2 either, keeps giving an “unknown filesystem” error. I’d jimmy’d a fix previously in 21.2, so was OK.

Sorry I don’t have time to track the issues down for now, I leave it to the more qualified folk on the forum.

Thanks.

1 Like
2

I use systemd-boot and this issue also occurred on my system. After rebooting, the system hang at rootfs prompt, with errors saying pool was previously used in another system. I used

zpool import -f POOLNAME

And tried to reboot, resulting in kernel panic. I pressed reset button on my computer, and it booted into system again. Maybe I was lucky.

3

One of our packagers tried to adopt changes from the AUR, which broke also encrypted systems. I reverted it back to a known working state of the PKGBUILD and files. I hope [pkg-upd] 2.1.2-2 (658d586e) · Commits · Packages / Extra / zfs-utils · GitLab fixes it …

1 Like
6

I got bit by this last week. I was able to figure out the hook had been changed; there was no prompt for password to load the ZFS key. I managed to boot into a USB rescue image and then chroot to modify the zfs hook:

[root@rescue mkinitcpio.d]# diff /usr/lib/initcpio/hooks/zfs /usr/lib/initcpio/hooks/zfs.orig
76,78d75
<                 echo "loading zfs key(s)"
<                 zfs load-key -a

After mkinitcpio --verbose --allpresets, i was able to decompress the verify the initramfs:

zcat initramfs-5.15-x86_64.img | cpio -idmv

And verified that the hooks/zfs script had my change in it to prompt for a password. Unmount / exit the rescue boot and was able to boot back into my regular system.

However, I am now stuck again after running / installing updates.

Like @dobedobedo I am getting a similar prompt.

ZFS: importing pool rpool_abcdef
cannot import 'rpool_abcdef': pool was previously in use from another system.
Last accessed by $hostFQDN at ...
The ppl can be imported, use `zpool import -f` to import the pool.

And then a kernel panic: not syncing: Attempted to kill init!

Unfortunately, I am not getting dropped into a recovery / single user shell. Adding init=/bin/bash to the grub boot entry does not drop me to a shell, same with systemd.unit=emergency.target

Any ideas on how I can run zfs import -f from the init boot env or otherwise fix this?

7

I spent an hour searching before posting… so of COURSE I find a solution after posting.

Hold shift to get grub, edit the boot options and add zfs_force=1 to the kernel params.