Failure to boot Manjaro from legacy USB with BTRFS after update mkinitcpio-35.2-2 with manjaro 22.1.0-1

I carry around a Linux live USB stick on my keychain as well, and it’s crazy how handy it’s been.

As far as I know, you can’t do a full install to a USB stick then expect it to boot on different computers, at least with any sort of stability.

This is why you need the live boot image to be able to boot up on any type of computer. It does a whole different boot up sequence for detecting hardware. Sure it takes a bit longer, but if you are booting a Linux install on different computers, you have to do it this way.

I wanted the exact same thing, for my changes to persist. From doing updates, to saving my @home volume, etc… So I’ve now been doing this for many years. I believe there are distros, and even Windows image tools that can do this for you, Rufus, Pendrive? Not sure actually, I do this all in Linux myself manually.

But the key is using casper-rw. Essentially, the Live environment stays the same and is readonly essentially, but any changes you make to it, get appended to a different area on the drive. But you are always seeing the latest on what you’ve changed. Whether it’s system updates, or documents and settings in your home folder. This does cause many updates or similar things cause it to grow and grow and grow. This is very similar to how a virtual sparse disk works for virtualization, rather than a pre-allocated virtual disk.

Though I haven’t done it personally, I know you can essentially shrink the changes over time when it starts getting too big and holding a lot of redundant or unneeded data.

Have you ever used alma ? :mag:

[manjaro /]# cat /etc/fstab
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a device; this may
# be used with UUID= as a more robust way to name devices that works even if
# disks are added and removed. See fstab(5).
#
# <file system>             <mount point>  <type>  <options>  <dump>  <pass>
UUID=7f813052-d472-4b7c-ad3c-65678589d9fa /              btrfs   subvol=/@,defaults 0 0
UUID=7f813052-d472-4b7c-ad3c-65678589d9fa /home          btrfs   subvol=/@home,defaults 0 0
UUID=7f813052-d472-4b7c-ad3c-65678589d9fa /var/cache     btrfs   subvol=/@cache,defaults 0 0
UUID=7f813052-d472-4b7c-ad3c-65678589d9fa /var/log       btrfs   subvol=/@log,defaults 0 0
$ lsblk -f
NAME   FSTYPE   FSVER            LABEL             UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
loop0  squashfs 4.0                                                                           0   100% /run/miso/sfs/livefs
loop1  squashfs 4.0                                                                           0   100% /run/miso/sfs/mhwdfs
loop2  squashfs 4.0                                                                           0   100% /run/miso/sfs/desktopfs
loop3  squashfs 4.0                                                                           0   100% /run/miso/sfs/rootfs
sda    iso9660  Joliet Extension MANJARO_XFCE_2205 2023-03-16-12-29-59-00                     0   100% /run/miso/bootmnt
├─sda1 iso9660  Joliet Extension MANJARO_XFCE_2205 2023-03-16-12-29-59-00                              
└─sda2 vfat     FAT12            MISO_EFI          5925-7716                                           
sdb                                                                                                    
└─sdb1 btrfs                                       7f813052-d472-4b7c-ad3c-65678589d9fa   15,7G    46% /mnt/home
                                                                                                       /mnt/var/log
                                                                                                       /mnt/var/cache
                                                                                                       /mnt

My problem seems to be with mkinitcpio:
I ran the whole story again:

  • Boot from Installation media
  • Install to another USB, legacy, MBR, BTRFS
  • Boot from the new installation
  • Install linux 6.2, remove onlyoffice
  • save a Timeshift snapshot
  • Pamac load german mirrorlist, update database,
  • install updates - except mkinitcpio(!)
  • reboot
  • save another Timeshift snapshot
  • Pamac install mkinitcpio 35.2-2 over the existing 34.1-1
    (This also re-creates the initcpio-images - after the update!)
  • mount: /new_root can’t find UUID=… (scrambled font view)

So to get my ‘old’ USB installations booting again, it seems, I’d just need to downgrade mkinitcpio and run it. But there I have another problem that I don’t know how to solve:

[manjaro /]# env DOWNGRADE_FROM_ALA=1 downgrade mkinitcpio
:: Retrieving packages...
 mkinitcpio-34-2-any    47.4 KiB   246 KiB/s 00:00 [######################] 100%
loading packages...
warning: downgrading package mkinitcpio (35.2-2 => 34-2)
resolving dependencies...
looking for conflicting packages...

Packages (1) mkinitcpio-34-2

Total Installed Size:   0.12 MiB
Net Upgrade Size:      -0.01 MiB

:: Proceed with installation? [Y/n] 
(1/1) checking keys in keyring                     [######################] 100%
(1/1) checking package integrity                   [######################] 100%
(1/1) loading package files                        [######################] 100%
(1/1) checking for file conflicts                  [######################] 100%
(1/1) checking available disk space                [######################] 100%
:: Running pre-transaction hooks...
(1/1) Creating Timeshift snapshot before upgrade...
E: System disk not found!
Unable to run timeshift-autosnap! Please close Timeshift and try again. Script will now exit...
error: command failed to execute correctly
error: failed to commit transaction (failed to run transaction hooks)
Errors occurred, no packages were upgraded.

(Chroot-ed mount on Installation boot media.)
On this freshly installed system I have the Timeshift snapshot and can go back to a bootable state, but since I couldn’t find out how to downgrade mkinitcpio each following system update will use 35.2-2 and render my system un-bootable.

What can I do to get back to the older mkinitcpio-34.1-1? How can I disable the timeshift-autosnap during this procedure, since it seems to be part of the package script?

D.

Update:
Added actual lsblk -f in this situation. D.

These UUIDs do not match UUID from the output of lsblk -f

  1. Chroot to your system via USB stick:
$ mount /dev/sda1 -o subvol=@       /mnt
$ mount /dev/sda1 -o subvol=@log    /mnt/var/log
$ mount /dev/sda1 -o subvol=@cache  /mnt/var/cache
$ manjaro-chroot /mnt
# nano /etc/fstab
  1. Try to change these UUIDs to 5ecd931a-e95c-44f3-ba9d-bbcf5ca4b0f1 in /etc/fstab

  2. Then run pacman -Syu && mkinitcpio -P && update-grub

That UUID was from an earlier installation attempt - sorry - upon every installation the filesystem gets a new one, of course.

Please, see above added lsblk output.

D.

Try to run SKIP_AUTOSNAP=1 env DOWNGRADE_FROM_ALA=1 downgrade mkinitcpio to skip/ignore timeshift-autosnap.

Yes, that seems to resolve my primary problem - my fresh USB installation is booting again after

# pacman -Sy downgrade
# env SKIP_AUTOSNAP=1 DOWNGRADE_FROM_ALA=1 downgrade mkinitcpio

and selecting mkinitcpio-34-2-any.

Now perhaps I have to report upstream that 35.2 breaks the boot from BTRFS?

Or should I stick with mkinitcpio-34? Perhaps, that’s not safe on the long term.
On the other hand, if the issue will not be fixed, I’ll have to do new installations on ext4 …
Just for the records: I’ve been using and updating the systems installed on USB happily on both computers alternatively for several years now (at least since 2019).

Many thanks,
Dirk

I can not reproduce this issue with mkinitcpio v35.2 in legacy BIOS in KVM. :man_shrugging:

Can you show us:

  • $ sudo btrfs subvolume list /
  • $ cat /etc/mkinitcpio.conf
  • $ cat /etc/default/grub

From my fresh blank installation (except that I now downgraded mkinitcpio):

$ sudo btrfs subvolume list /

ID 256 gen 206 top level 5 path @
ID 257 gen 209 top level 5 path @home
ID 258 gen 205 top level 5 path @cache
ID 259 gen 208 top level 5 path @log
ID 260 gen 79 top level 5 path timeshift-btrfs/snapshots/2023-04-02_11-13-57/@
ID 261 gen 89 top level 5 path timeshift-btrfs/snapshots/2023-04-02_11-20-10/@
ID 262 gen 98 top level 5 path timeshift-btrfs/snapshots/2023-04-02_11-24-49/@
ID 263 gen 123 top level 5 path timeshift-btrfs/snapshots/2023-04-02_11-38-15/@
ID 264 gen 126 top level 5 path timeshift-btrfs/snapshots/2023-04-02_11-40-01/@
ID 265 gen 171 top level 5 path timeshift-btrfs/snapshots/2023-04-02_13-26-03/@
$ cat /etc/mkinitcpio.conf
# vim:set ft=sh
# MODULES
# The following modules are loaded before any boot hooks are
# run.  Advanced users may wish to specify all system modules
# in this array.  For instance:
#     MODULES=(usbhid xhci_hcd)
MODULES="crc32c-intel"

# BINARIES
# This setting includes any additional binaries a given user may
# wish into the CPIO image.  This is run last, so it may be used to
# override the actual binaries included by a given hook
# BINARIES are dependency parsed, so you may safely ignore libraries
BINARIES=()

# FILES
# This setting is similar to BINARIES above, however, files are added
# as-is and are not parsed in any way.  This is useful for config files.
FILES=""

# HOOKS
# This is the most important setting in this file.  The HOOKS control the
# modules and scripts added to the image, and what happens at boot time.
# Order is important, and it is recommended that you do not change the
# order in which HOOKS are added.  Run 'mkinitcpio -H <hook name>' for
# help on a given hook.
# 'base' is _required_ unless you know precisely what you are doing.
# 'udev' is _required_ in order to automatically load modules
# 'filesystems' is _required_ unless you specify your fs modules in MODULES
# Examples:
##   This setup specifies all modules in the MODULES setting above.
##   No RAID, lvm2, or encrypted root is needed.
#    HOOKS=(base)
#
##   This setup will autodetect all modules for your system and should
##   work as a sane default
#    HOOKS=(base udev autodetect modconf block filesystems fsck)
#
##   This setup will generate a 'full' image which supports most systems.
##   No autodetection is done.
#    HOOKS=(base udev modconf block filesystems fsck)
#
##   This setup assembles a mdadm array with an encrypted root file system.
##   Note: See 'mkinitcpio -H mdadm_udev' for more information on RAID devices.
#    HOOKS=(base udev modconf keyboard keymap consolefont block mdadm_udev encrypt filesystems fsck)
#
##   This setup loads an lvm2 volume group.
#    HOOKS=(base udev modconf block lvm2 filesystems fsck)
#
##   NOTE: If you have /usr on a separate partition, you MUST include the
#    usr and fsck hooks.
HOOKS="base udev autodetect modconf block keyboard keymap consolefont plymouth filesystems"

# COMPRESSION
# Use this to compress the initramfs image. By default, gzip compression
# is used. Use 'cat' to create an uncompressed image.
#COMPRESSION="gzip"
#COMPRESSION="bzip2"
#COMPRESSION="lzma"
#COMPRESSION="xz"
#COMPRESSION="lzop"
#COMPRESSION="lz4"
#COMPRESSION="zstd"

# COMPRESSION_OPTIONS
# Additional options for the compressor
#COMPRESSION_OPTIONS=()

# MODULES_DECOMPRESS
# Decompress kernel modules during initramfs creation.
# Enable to speedup boot process, disable to save RAM
# during early userspace. Switch (yes/no).
#MODULES_DECOMPRESS="yes"
$ cat /etc/default/grub
GRUB_DEFAULT=saved
GRUB_TIMEOUT=5
GRUB_TIMEOUT_STYLE=hidden
GRUB_DISTRIBUTOR="Manjaro"
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash apparmor=1 security=apparmor udev.log_priority=3"
GRUB_CMDLINE_LINUX=""

# If you want to enable the save default function, uncomment the following
# line, and set GRUB_DEFAULT to saved.
#GRUB_SAVEDEFAULT="true"

# Uncomment to disable submenus in boot menu
#GRUB_DISABLE_SUBMENU=y

# Preload both GPT and MBR modules so that they are not missed
GRUB_PRELOAD_MODULES="part_gpt part_msdos"

# Uncomment to enable booting from LUKS encrypted devices
#GRUB_ENABLE_CRYPTODISK=y

# Uncomment to use basic console
GRUB_TERMINAL_INPUT=console

# Uncomment to disable graphical terminal
#GRUB_TERMINAL_OUTPUT=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command 'videoinfo'
GRUB_GFXMODE=auto

# Uncomment to allow the kernel use the same resolution used by grub
GRUB_GFXPAYLOAD_LINUX=keep

# Uncomment if you want GRUB to pass to the Linux kernel the old parameter
# format "root=/dev/xxx" instead of "root=/dev/disk/by-uuid/xxx"
#GRUB_DISABLE_LINUX_UUID=true

# Uncomment to disable generation of recovery mode menu entries
GRUB_DISABLE_RECOVERY=true

# Uncomment this option to enable os-prober execution in the grub-mkconfig command
GRUB_DISABLE_OS_PROBER=false

# Uncomment and set to the desired menu colors.  Used by normal and wallpaper
# modes only.  Entries specified as foreground/background.
GRUB_COLOR_NORMAL="light-gray/black"
GRUB_COLOR_HIGHLIGHT="green/black"

# Uncomment one of them for the gfx desired, a image background or a gfxtheme
#GRUB_BACKGROUND="/usr/share/grub/background.png"
GRUB_THEME="/usr/share/grub/themes/manjaro/theme.txt"

# Uncomment to get a beep at GRUB start
#GRUB_INIT_TUNE="480 440 1"

# Uncomment to ensure that the root filesystem is mounted read-only so that
# systemd-fsck can run the check automatically. We use 'fsck' by default, which
# needs 'rw' as boot parameter, to avoid delay in boot-time. 'fsck' needs to be
# removed from 'mkinitcpio.conf' to make 'systemd-fsck' work.
# See also Arch-Wiki: https://wiki.archlinux.org/index.php/Fsck#Boot_time_checking
#GRUB_ROOT_FS_RO=true

On my second old system (which boots again :slight_smile: ) the installer had created only a separate home subvolume then:

$ sudo btrfs subvolume list /

ID 256 gen 407793 top level 5 path @
ID 258 gen 407793 top level 5 path @home

There are some old or bad configurations:

Change

to

MODULES=(crc32c-intel)

Remove consolefont in HOOKS:
Change

to

HOOKS=(base udev autodetect modconf block keyboard keymap plymouth filesystems)#

If you use Kernel 6.2, enable zstd for fast compression and fast decompression at boot. Do not use Kernel older than 5.10 LTS
Change

to

COMPRESSION="zstd"

If you use grub-btrfs, change

to

GRUB_DEFAULT=0

Then run $ sudo mkinitcpio -P and $ sudo update-grub

Hi all, new to manjaro forums, but not to manjaro (using it for the last 5 years). This weekend I faced the same problem after updating, and what solved the problem for me was downgrading mkinitcpio to 34.2. I don’t have BTRFS but EXT4. The only thing I have in common is that my Manjaro installation is in an external SSD connected by USB.

I tried to change kernel, to replace grub by refind and some more checks, but in everyone I did, I faced the same problem. Only downgrading mkinitcpio solved the problem for me. Just trying to give more info to this issue.

If you need any info about my config, I will be glad to help.

1 Like

Do you use UEFI or legacy BIOS?

That’s interesting. As described before, the system containing these files was freshly installed from “manjaro-xfce-22.0.5-230316-linux61.iso” downloaded yesterday, dd-ed to USB; then just updated to 22.1.0-1 by pamac. If there are some configurations bad or old, this should be addressed to the distribution image also. I’ll follow your notes and then try with mkinitcpio 35.2-2 again.
D.

Result: After applying the changes to the files and installing the mkinitcpio upgrade (which involves the mkinitcpio and update-grub), the (fresh) system again failed to boot (message font scrambled as before - I had tried VGA instead of HDMI yesterday, still not readable; but judging from the structure, it’s the same error message for the actual uuid). D.

There’s another report of a user who also boots from exernal USB, keeping the internal drives unchanged:
https://forum.manjaro.org/t/cinnamon-flash-usb-stops-after-grub-after-most-recent-update-today-morning/137458/20

Did you check other version 35.1 (not 35.2) of mkinitcpio if it has the same problem?

No, I did not try that. When I stuck on that problem it was Saturday morning and pamac just showed that it would update 34.1-1 to 35.2-2. I can try. D.

Downgraded my ‘fresh’ (stable) installation to mkinitcpio-35.1-1 and stuck with the same scrambled message - very likely mount /new_root couldn’t find uuid=…
D.

2 Likes

UEFI, and my Manjaro installation is completely isolated. I mean, it has its own boot partition with everything needed to run on any computer. So I’m not sharing GRUB to run Windows, manjaro or any other OS. Each OS has its own boot manager/EFI partition, and I select each OS by selecting the device once I power on the computer.

With the update to mkinitcpio 36-2 the issue is resolved for me - after the update an update-grub ran automatically, and all my different usb installations (manjaro stable xfce btrfs on usb) do boot after this.

From today’s struggle with pamac I learned that certain updates may create files with extension .pacnew (mostly in /etc and subdirs), which may contain settings that need to be added or updated into the original configuration files like /etc/pacman.conf or /etc/default/grub e.g. if the update routine cannot handle this automatically.

1 Like