Failure to boot Manjaro from legacy USB with BTRFS after update mkinitcpio-35.2-2 with manjaro 22.1.0-1

ddeberhar · 1 April 2023 18:07

After running updates today 2023-04-01 (no joke!) including grub and btrfs-tools on my live system that I installed long ago from Manjaro XFCE installation media to a BTRFS partition on an USB media, now I get upon boot attempt:

mount: /new_root: cant't find UUID=0da7e ....
You are now being dropped into an emergency shell.
sh: can't access tty: job control turned off
[rootfs ]# _

This happened on two differend USB installations on two different computers, both installations were up to date a week ago. On one of them I disbled the disk controller, so that the USB media became /dev/sda (if it had been found).

I have a third installation on an internal SSD (AHCI), also legacy on one BTRFS partition, that does not suffer from this issue - that’s the only one that I can boot now.

For verification I ran a fresh installation from “manjaro-xfce-22.0.5-230316-linux61.iso” on a new USB media again into one BTRFS partition, booted, ran the updates - et voilá … Same fault, only in scrambled font via HDMI graphics.

From the wiki I had formerly copied an instruction to reinstall GRUB, I tried to follow this (yet pacman -S mtools os-prober fails with a hint to restart Timeshift; I don’t know how to handle the Timeshift-error “system disk not found”), grub-install --recheck and update-grub gave no errors, the latter listed all kernels, but neither fixed my issue.

Next I’ll do a fresh installation on ext4 - if that survives the update, at least I’ll know.

Any other supporting hints heartily welcome.

Dirk

Update:
The same procedure - installing from XFCE-ISO to USB, just one difference - formatted the target to ext4 instead of btrfs - then deinstalled office (to save time during update), ran pamac with actual updates - gave the result you’d expect: System booting to login prompt. So the issue is that after the update the new (yet updated) kernel (6.1 from fresh installation, 6.2 or 5.15 from grub menu on my previous, even fallback) and ramdisk fail to recognize the UUID of the BTRFS filesystem. I checked while booted from the live installer - in thunar’s address line exactly the same uuid is visible for the volume in question.
My choice was for btrfs because of the copy-on-write feature - I thought, including wear-levelling the whole would be more reliable…
D.

brahma · 1 April 2023 18:23

chroot, rerun update:
pacman-mirrors --fasttrack 5 && pacman -Syyu
then this:
mkinitcpio -P && update-grub
sync
exit chroot, reboot and see if it helped

Molski · 1 April 2023 18:27

So you have 3 installations but the 2 installations that stopped booting are booting from a USB flash drive or whatever? I’m a little confused…

What’s the output of lslbk -f?

Does the UUID match what grub.cfg is trying to boot?

ddeberhar · 1 April 2023 23:09

Starting from the end - here’s the lsblk -f for my fresh installation:

$ lsblk -f
NAME FSTYPE FSVER LABEL UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
sda                                                                         
└─sda1
     btrfs              5ecd931a-e95c-44f3-ba9d-bbcf5ca4b0f1   20,6G    29% /var/log
                                                                            /var/cache
                                                                            /home
                                                                            /

Unfortunately I cannot really confirm whether the error message after the update matches the UUID:

But it looks very similiar to what I got from my old USB installation, which behaves a little different at this point:

This UUID matches with what I see if I boot a functional system and attach the USB media with this filesystem. Filesystem mountable and accessible without any issue.

Now about my experiments with the fresh installation - again on BTRFS - after the installation on EXT4 did not exhibit any problems after the update (mentioned in my updated report on top).
The failure to locate the root filesystem appeared again after running the 319 packages update via pamac.
No attempts to omit grub or btrfs-tools (after recovering from a pre-update snapshot and updating again) changed that.
Unfortunately my old live USB installations do not offer any snapshots menu entry in the GRUB menus - just minutes ago pamac listted a new update to mkinitcpio; I’ll test that first on my fresh USB installation …

A little history explanation:
After I found it taking too much time to create a customized Knoppix after each Firefox update, I looked for a rolling release distro and moved to Manjaro.
I continued with an installation on a USB stick that I could use as well in my notebook as in my desktop as required.
Then I made a second USB installation for the case my first one would fail, and finally I installed on the internal SSD of my desktop.
This time I started with the desktop update - as it did not show any issues, I ran the USB updates at the same time and ended with two installations not booting.

D.

Molski · 2 April 2023 03:45

I do carry around a tiny flash fob on my key chain that can I live boot in any computer, but there is a not so simple trick to have the data persistent so you can update it and whatnot.

But you do this for the same computer? I’m still trying to understand your use case and why you are even doing this in the first place. As a backup?

Either way, when troubleshooting issues like this, you would have to boot the Manjaro live/installer image. That is where you do the commands like lsblk -f. Then you mount your installation/drive (and manjaro-chroot into your environment if necessary).

ddeberhar · 2 April 2023 08:17

Why on USB: During daily (mobile) work I carry around a (work) notebook where I don’t want to touch the installed system on encrypted disk at all. So I came to carrying my personal system and data on a bootable USB-stick - just for the case that I might need it - which I can boot also at home on my desktop with 24" monitor, that I obviously cannot carry around all time; a third computer (second notebook) would just add to the burden…
My job is to repair failing computers, so I see a lot of faulty components, namely disks and SSDs - and I decided to maintain a second bootable system on USB, yet not with the data, for troubleshooting and possibly repair e.g.

Booting the installation media is a little time consuming: The included GRUB does not really like my desktop Asrock Q1900B-ITX with Celeron J1900 and i915 graphics at a Philips HDMI TV - it paints very slowly giant letters of few text lines and takes ages to respond to keyboard input, re-drawing everything upon each key pressed. Thus each and every time - set the timezone, system clock back to UTC, keyboard, locale, becomes somewhat boring. (The new splash screen looks also weird - after installation.)

Finally my desktop has only six USB ports available (only one of them USB 3.0), two are in use for keyboard and mouse; if a third is for the system, sometimes I feel reminded old times when I played ‘disk jockey’ upon a floppy disk installation, if I want to sort data on multiple media… That’s why the third installation on the internal SSD.

Anyway, I decided to troubleshoot in a reproducable environment, downloaded the most recent installation ISO, created a USB boot media, installed from there to BTRFS, legacy boot, MBR only, on a fresh blank USB media (in fact a micro SD with adapter in a 1-slot card reader), booted, removed the 8xx MB onlyoffice to speed the update a little. (Internal Storage disabled in BIOS.)

Upon the last attempt last night I even installed the newest kernel 6.2 first, rebooted, and then had pamac install the 1 GB 300+ package updates. On my two ‘old’ USB installations I had not observed any snapshot items in the GRUB menu, but they produce the mount error in a readable fashion. Now there are snapshots; I could recover the last bootable, update again, leaving out once grub, once btrfs-tools - no change, after the reboot ‘mount’ cannot find the root volume. As said above, I did all this once more, choosing just ext4 instead of btrfs, and had no trouble booting after the updates.

After my first report another mkinitcpio update had arrived, I installed that on my second ‘old’ USB via chroot (booted from installation ISO USB) - still the same problem, although update-grub had listed all kernels and fallback-options.

Now I’ll do one more attempt, leaving out mkinitcpio from the updates (if it’s there), but then I’m at the end of my wits. Perhaps I’ll have to install everything again, just on ext4, if I still wish to boot an up-to-date XFCE-Manjaro from USB.
(Formerly I had somewhat different plans for this weekend…)

As I’m not familiar with GRUB and the whole boot process (as well as how to get a copied system bootable, or mmcblk0, or NVMe), I can only guess that “mount: /new_root” may be issued already from the initial ramdisk - or is that message from GRUB?

D.

Zesko · 2 April 2023 08:56

Chroot to your system via USB stick first, then can you show us:

$ cat /etc/fstab

Wollie · 2 April 2023 09:20

I would rather think of using an external SSD drive than an USB stick, it’s a much better approach.
Also, make sure OS prober is disabled. If you proceed trying to use an USB, I would recommend this article:

Also worth reading:

Molski · 2 April 2023 09:20

I carry around a Linux live USB stick on my keychain as well, and it’s crazy how handy it’s been.

As far as I know, you can’t do a full install to a USB stick then expect it to boot on different computers, at least with any sort of stability.

This is why you need the live boot image to be able to boot up on any type of computer. It does a whole different boot up sequence for detecting hardware. Sure it takes a bit longer, but if you are booting a Linux install on different computers, you have to do it this way.

I wanted the exact same thing, for my changes to persist. From doing updates, to saving my @home volume, etc… So I’ve now been doing this for many years. I believe there are distros, and even Windows image tools that can do this for you, Rufus, Pendrive? Not sure actually, I do this all in Linux myself manually.

But the key is using casper-rw. Essentially, the Live environment stays the same and is readonly essentially, but any changes you make to it, get appended to a different area on the drive. But you are always seeing the latest on what you’ve changed. Whether it’s system updates, or documents and settings in your home folder. This does cause many updates or similar things cause it to grow and grow and grow. This is very similar to how a virtual sparse disk works for virtualization, rather than a pre-allocated virtual disk.

Though I haven’t done it personally, I know you can essentially shrink the changes over time when it starts getting too big and holding a lot of redundant or unneeded data.

andreas85 · 2 April 2023 10:09

Have you ever used alma ?

ddeberhar · 2 April 2023 10:26

[manjaro /]# cat /etc/fstab
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a device; this may
# be used with UUID= as a more robust way to name devices that works even if
# disks are added and removed. See fstab(5).
#
# <file system>             <mount point>  <type>  <options>  <dump>  <pass>
UUID=7f813052-d472-4b7c-ad3c-65678589d9fa /              btrfs   subvol=/@,defaults 0 0
UUID=7f813052-d472-4b7c-ad3c-65678589d9fa /home          btrfs   subvol=/@home,defaults 0 0
UUID=7f813052-d472-4b7c-ad3c-65678589d9fa /var/cache     btrfs   subvol=/@cache,defaults 0 0
UUID=7f813052-d472-4b7c-ad3c-65678589d9fa /var/log       btrfs   subvol=/@log,defaults 0 0

$ lsblk -f
NAME   FSTYPE   FSVER            LABEL             UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
loop0  squashfs 4.0                                                                           0   100% /run/miso/sfs/livefs
loop1  squashfs 4.0                                                                           0   100% /run/miso/sfs/mhwdfs
loop2  squashfs 4.0                                                                           0   100% /run/miso/sfs/desktopfs
loop3  squashfs 4.0                                                                           0   100% /run/miso/sfs/rootfs
sda    iso9660  Joliet Extension MANJARO_XFCE_2205 2023-03-16-12-29-59-00                     0   100% /run/miso/bootmnt
├─sda1 iso9660  Joliet Extension MANJARO_XFCE_2205 2023-03-16-12-29-59-00                              
└─sda2 vfat     FAT12            MISO_EFI          5925-7716                                           
sdb                                                                                                    
└─sdb1 btrfs                                       7f813052-d472-4b7c-ad3c-65678589d9fa   15,7G    46% /mnt/home
                                                                                                       /mnt/var/log
                                                                                                       /mnt/var/cache
                                                                                                       /mnt

My problem seems to be with mkinitcpio:
I ran the whole story again:

Boot from Installation media
Install to another USB, legacy, MBR, BTRFS
Boot from the new installation
Install linux 6.2, remove onlyoffice
save a Timeshift snapshot
Pamac load german mirrorlist, update database,
install updates - except mkinitcpio(!)
reboot
save another Timeshift snapshot
Pamac install mkinitcpio 35.2-2 over the existing 34.1-1
(This also re-creates the initcpio-images - after the update!)
mount: /new_root can’t find UUID=… (scrambled font view)

So to get my ‘old’ USB installations booting again, it seems, I’d just need to downgrade mkinitcpio and run it. But there I have another problem that I don’t know how to solve:

[manjaro /]# env DOWNGRADE_FROM_ALA=1 downgrade mkinitcpio
:: Retrieving packages...
 mkinitcpio-34-2-any    47.4 KiB   246 KiB/s 00:00 [######################] 100%
loading packages...
warning: downgrading package mkinitcpio (35.2-2 => 34-2)
resolving dependencies...
looking for conflicting packages...

Packages (1) mkinitcpio-34-2

Total Installed Size:   0.12 MiB
Net Upgrade Size:      -0.01 MiB

:: Proceed with installation? [Y/n] 
(1/1) checking keys in keyring                     [######################] 100%
(1/1) checking package integrity                   [######################] 100%
(1/1) loading package files                        [######################] 100%
(1/1) checking for file conflicts                  [######################] 100%
(1/1) checking available disk space                [######################] 100%
:: Running pre-transaction hooks...
(1/1) Creating Timeshift snapshot before upgrade...
E: System disk not found!
Unable to run timeshift-autosnap! Please close Timeshift and try again. Script will now exit...
error: command failed to execute correctly
error: failed to commit transaction (failed to run transaction hooks)
Errors occurred, no packages were upgraded.

(Chroot-ed mount on Installation boot media.)
On this freshly installed system I have the Timeshift snapshot and can go back to a bootable state, but since I couldn’t find out how to downgrade mkinitcpio each following system update will use 35.2-2 and render my system un-bootable.

What can I do to get back to the older mkinitcpio-34.1-1? How can I disable the timeshift-autosnap during this procedure, since it seems to be part of the package script?

D.

Update:
Added actual lsblk -f in this situation. D.

Zesko · 2 April 2023 10:41

ddeberhar:

# <file system>             <mount point>  <type>  <options>  <dump>  <pass>
UUID=7f813052-d472-4b7c-ad3c-65678589d9fa /              btrfs   subvol=/@,defaults 0 0
UUID=7f813052-d472-4b7c-ad3c-65678589d9fa /home          btrfs   subvol=/@home,defaults 0 0
UUID=7f813052-d472-4b7c-ad3c-65678589d9fa /var/cache     btrfs   subvol=/@cache,defaults 0 0
UUID=7f813052-d472-4b7c-ad3c-65678589d9fa /var/log       btrfs   subvol=/@log,defaults 0 0

These UUIDs do not match UUID from the output of lsblk -f

ddeberhar:

$ lsblk -f
NAME FSTYPE FSVER LABEL UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
sda                                                                         
└─sda1
     btrfs              5ecd931a-e95c-44f3-ba9d-bbcf5ca4b0f1   20,6G    29% /var/log
                                                                            /var/cache
                                                                            /home
                                                                            /

Chroot to your system via USB stick:

$ mount /dev/sda1 -o subvol=@       /mnt
$ mount /dev/sda1 -o subvol=@log    /mnt/var/log
$ mount /dev/sda1 -o subvol=@cache  /mnt/var/cache
$ manjaro-chroot /mnt
# nano /etc/fstab

Try to change these UUIDs to 5ecd931a-e95c-44f3-ba9d-bbcf5ca4b0f1 in /etc/fstab
Then run pacman -Syu && mkinitcpio -P && update-grub

ddeberhar · 2 April 2023 10:46

That UUID was from an earlier installation attempt - sorry - upon every installation the filesystem gets a new one, of course.

Please, see above added lsblk output.

D.

Zesko · 2 April 2023 10:54

Try to run SKIP_AUTOSNAP=1 env DOWNGRADE_FROM_ALA=1 downgrade mkinitcpio to skip/ignore timeshift-autosnap.

ddeberhar · 2 April 2023 11:30

Yes, that seems to resolve my primary problem - my fresh USB installation is booting again after

# pacman -Sy downgrade
# env SKIP_AUTOSNAP=1 DOWNGRADE_FROM_ALA=1 downgrade mkinitcpio

and selecting mkinitcpio-34-2-any.

Now perhaps I have to report upstream that 35.2 breaks the boot from BTRFS?

Or should I stick with mkinitcpio-34? Perhaps, that’s not safe on the long term.
On the other hand, if the issue will not be fixed, I’ll have to do new installations on ext4 …
Just for the records: I’ve been using and updating the systems installed on USB happily on both computers alternatively for several years now (at least since 2019).

Many thanks,
Dirk

Zesko · 2 April 2023 11:44

I can not reproduce this issue with mkinitcpio v35.2 in legacy BIOS in KVM.

Can you show us:

$ sudo btrfs subvolume list /
$ cat /etc/mkinitcpio.conf
$ cat /etc/default/grub

ddeberhar · 2 April 2023 12:15

From my fresh blank installation (except that I now downgraded mkinitcpio):

$ sudo btrfs subvolume list /

ID 256 gen 206 top level 5 path @
ID 257 gen 209 top level 5 path @home
ID 258 gen 205 top level 5 path @cache
ID 259 gen 208 top level 5 path @log
ID 260 gen 79 top level 5 path timeshift-btrfs/snapshots/2023-04-02_11-13-57/@
ID 261 gen 89 top level 5 path timeshift-btrfs/snapshots/2023-04-02_11-20-10/@
ID 262 gen 98 top level 5 path timeshift-btrfs/snapshots/2023-04-02_11-24-49/@
ID 263 gen 123 top level 5 path timeshift-btrfs/snapshots/2023-04-02_11-38-15/@
ID 264 gen 126 top level 5 path timeshift-btrfs/snapshots/2023-04-02_11-40-01/@
ID 265 gen 171 top level 5 path timeshift-btrfs/snapshots/2023-04-02_13-26-03/@

$ cat /etc/mkinitcpio.conf
# vim:set ft=sh
# MODULES
# The following modules are loaded before any boot hooks are
# run.  Advanced users may wish to specify all system modules
# in this array.  For instance:
#     MODULES=(usbhid xhci_hcd)
MODULES="crc32c-intel"

# BINARIES
# This setting includes any additional binaries a given user may
# wish into the CPIO image.  This is run last, so it may be used to
# override the actual binaries included by a given hook
# BINARIES are dependency parsed, so you may safely ignore libraries
BINARIES=()

# FILES
# This setting is similar to BINARIES above, however, files are added
# as-is and are not parsed in any way.  This is useful for config files.
FILES=""

# HOOKS
# This is the most important setting in this file.  The HOOKS control the
# modules and scripts added to the image, and what happens at boot time.
# Order is important, and it is recommended that you do not change the
# order in which HOOKS are added.  Run 'mkinitcpio -H <hook name>' for
# help on a given hook.
# 'base' is _required_ unless you know precisely what you are doing.
# 'udev' is _required_ in order to automatically load modules
# 'filesystems' is _required_ unless you specify your fs modules in MODULES
# Examples:
##   This setup specifies all modules in the MODULES setting above.
##   No RAID, lvm2, or encrypted root is needed.
#    HOOKS=(base)
#
##   This setup will autodetect all modules for your system and should
##   work as a sane default
#    HOOKS=(base udev autodetect modconf block filesystems fsck)
#
##   This setup will generate a 'full' image which supports most systems.
##   No autodetection is done.
#    HOOKS=(base udev modconf block filesystems fsck)
#
##   This setup assembles a mdadm array with an encrypted root file system.
##   Note: See 'mkinitcpio -H mdadm_udev' for more information on RAID devices.
#    HOOKS=(base udev modconf keyboard keymap consolefont block mdadm_udev encrypt filesystems fsck)
#
##   This setup loads an lvm2 volume group.
#    HOOKS=(base udev modconf block lvm2 filesystems fsck)
#
##   NOTE: If you have /usr on a separate partition, you MUST include the
#    usr and fsck hooks.
HOOKS="base udev autodetect modconf block keyboard keymap consolefont plymouth filesystems"

# COMPRESSION
# Use this to compress the initramfs image. By default, gzip compression
# is used. Use 'cat' to create an uncompressed image.
#COMPRESSION="gzip"
#COMPRESSION="bzip2"
#COMPRESSION="lzma"
#COMPRESSION="xz"
#COMPRESSION="lzop"
#COMPRESSION="lz4"
#COMPRESSION="zstd"

# COMPRESSION_OPTIONS
# Additional options for the compressor
#COMPRESSION_OPTIONS=()

# MODULES_DECOMPRESS
# Decompress kernel modules during initramfs creation.
# Enable to speedup boot process, disable to save RAM
# during early userspace. Switch (yes/no).
#MODULES_DECOMPRESS="yes"

$ cat /etc/default/grub
GRUB_DEFAULT=saved
GRUB_TIMEOUT=5
GRUB_TIMEOUT_STYLE=hidden
GRUB_DISTRIBUTOR="Manjaro"
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash apparmor=1 security=apparmor udev.log_priority=3"
GRUB_CMDLINE_LINUX=""

# If you want to enable the save default function, uncomment the following
# line, and set GRUB_DEFAULT to saved.
#GRUB_SAVEDEFAULT="true"

# Uncomment to disable submenus in boot menu
#GRUB_DISABLE_SUBMENU=y

# Preload both GPT and MBR modules so that they are not missed
GRUB_PRELOAD_MODULES="part_gpt part_msdos"

# Uncomment to enable booting from LUKS encrypted devices
#GRUB_ENABLE_CRYPTODISK=y

# Uncomment to use basic console
GRUB_TERMINAL_INPUT=console

# Uncomment to disable graphical terminal
#GRUB_TERMINAL_OUTPUT=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command 'videoinfo'
GRUB_GFXMODE=auto

# Uncomment to allow the kernel use the same resolution used by grub
GRUB_GFXPAYLOAD_LINUX=keep

# Uncomment if you want GRUB to pass to the Linux kernel the old parameter
# format "root=/dev/xxx" instead of "root=/dev/disk/by-uuid/xxx"
#GRUB_DISABLE_LINUX_UUID=true

# Uncomment to disable generation of recovery mode menu entries
GRUB_DISABLE_RECOVERY=true

# Uncomment this option to enable os-prober execution in the grub-mkconfig command
GRUB_DISABLE_OS_PROBER=false

# Uncomment and set to the desired menu colors.  Used by normal and wallpaper
# modes only.  Entries specified as foreground/background.
GRUB_COLOR_NORMAL="light-gray/black"
GRUB_COLOR_HIGHLIGHT="green/black"

# Uncomment one of them for the gfx desired, a image background or a gfxtheme
#GRUB_BACKGROUND="/usr/share/grub/background.png"
GRUB_THEME="/usr/share/grub/themes/manjaro/theme.txt"

# Uncomment to get a beep at GRUB start
#GRUB_INIT_TUNE="480 440 1"

# Uncomment to ensure that the root filesystem is mounted read-only so that
# systemd-fsck can run the check automatically. We use 'fsck' by default, which
# needs 'rw' as boot parameter, to avoid delay in boot-time. 'fsck' needs to be
# removed from 'mkinitcpio.conf' to make 'systemd-fsck' work.
# See also Arch-Wiki: https://wiki.archlinux.org/index.php/Fsck#Boot_time_checking
#GRUB_ROOT_FS_RO=true

On my second old system (which boots again ) the installer had created only a separate home subvolume then:

$ sudo btrfs subvolume list /

ID 256 gen 407793 top level 5 path @
ID 258 gen 407793 top level 5 path @home

Zesko · 2 April 2023 12:29

There are some old or bad configurations:

Change

to

MODULES=(crc32c-intel)

Remove consolefont in HOOKS:
Change

to

HOOKS=(base udev autodetect modconf block keyboard keymap plymouth filesystems)#

If you use Kernel 6.2, enable zstd for fast compression and fast decompression at boot. Do not use Kernel older than 5.10 LTS
Change

to

COMPRESSION="zstd"

If you use grub-btrfs, change

to

GRUB_DEFAULT=0

Then run $ sudo mkinitcpio -P and $ sudo update-grub

Varo · 2 April 2023 12:50

Hi all, new to manjaro forums, but not to manjaro (using it for the last 5 years). This weekend I faced the same problem after updating, and what solved the problem for me was downgrading mkinitcpio to 34.2. I don’t have BTRFS but EXT4. The only thing I have in common is that my Manjaro installation is in an external SSD connected by USB.

I tried to change kernel, to replace grub by refind and some more checks, but in everyone I did, I faced the same problem. Only downgrading mkinitcpio solved the problem for me. Just trying to give more info to this issue.

If you need any info about my config, I will be glad to help.

Zesko · 2 April 2023 12:52

Do you use UEFI or legacy BIOS?