Sudden failure to boot

Aragorn · 5 June 2025 01:13

Hey People, this time I need your help — and badly too.

Here’s the thing… For a couple of weeks already, I had been brooding on converting my existing Manjaro installation with multiple distinct partitions — i.e. the usual EFI partition, an ext4 partition for /boot, a (disabled) swap partition, an unused (for too small) btrfs partition that used to be mounted at /var, and about 7 other distinct btrfs partitions — into a system with the following setup…:

/dev/sda1 – /boot/efi – vfat (unchanged)
/dev/sda2 – /boot – ext4 (unchanged)
/dev/sda3 – everything else — btrfs subvolumes (and a whole bunch of them too!)

Now, I have studied the Arch Wiki, I have read countless man pages — well, okay, maybe not “countless”, but many of them anyway — and I have even consulted duck.ai, the ChatGPT4 Mini of DuckDuckGo. And it was quite helpful on account of technical details that are often overlooked or ambiguously worded in the man pages or online information sources.

So I have done my due diligence, and I am pretty certain that I was also very meticulous and methodical in my efforts. However, I cannot really test my endeavor — pun not intended, although I will get back to this in a short while — because my system refuses to boot, with an error message that it cannot mount /boot, and an “Assertion” error about the initramfs.

Now, the thing is that before I went ahead with the conversion, I already had a similar failure to boot with my system as it was installed. And I had only really done two or three things, one of which threw an error.

The first thing I did was use gparted to give my first three partitions a PARTLABEL, and in addition to that, a filesystem LABEL for /dev/sda2 and /dev/sda3. At that point in time, /dev/sda3 already was my root filesystem, and it was btrfs, but it was only 1 GiB in size, and it initially only had one subvolume on it, “@”, created by calamares. My conversion afterwards was going to include the removal of all following partitions and the enlargement of /dev/sda3 to fill the rest of the drive.
The second thing I did was create additional (but empty) subvolumes on /dev/sda3.
The third thing I did — and which was the one that threw up an error, even though it had nothing to do with the above two actions — was update my system, because there was an update for chromium waiting, and to then, while I was at it, uninstall mkinitcpio-openswap. I don’t really know why I had that installed, and it was marked as “explicitly installed”. But given that I don’t even use a swap device — with 16 GiB of RAM, I haven’t needed any swap in almost 6 years now — I uninstalled it.

It was this latter action which triggered an automatic rebuild of my initramfs, and that’s where things suddenly went all haywire. Even though the initramfs appeared to have been correctly built — it reported success, and it showed me all of the hooks — there was an error message from zstd right after showing the [kms] hook, and it was repeatable. I’ve retried rebuilding the initramfs, and each time the error message — going from memory, it was something along the lines of “zstd error 70: broken pipe” — appeared right after the [kms] hook appeared on the screen.

I didn’t think much of it, given that the rebuild of the initramfs appeared to have succeeded well. So before doing anything else, I decided to download the latest Manjaro ISO for my Ventoy stick. But stupidly, after removing the older Manjaro ISO from the Ventoy stick, I had forgotten to move the new ISO from /tmp to the Ventoy stick – I always download ISOs to /tmp, which is faster than an SSD, and which spares the SSD a few write-and-erase cycles.

Therefore, upon rebooting from the Ventoy stick, I saw that I only still had Endeavour OS on there, which is what I’m using now, and it’s a pain in the nether regions, because it’s completely set up for laptops with US keyboards and WiFi, with a dimming screen, power savings all over the place, double-click to open instead of single-click, an incredibly slow mouse with acceleration disabled, and so on.

I literally have to spend at least half an hour tweaking all the settings — including changing the wallpaper to something that causes less bleeding in my eyes and enlarging the ant-sized fonts. It also takes ages to boot, because by default, it loads a firewall that slows everything down, kdeconnect, and something similar to manjaro-hello, but which also takes a long time to load. And, oh, yes, you’re on a slow power-savings profile by default, so you have to change that too.

Anyway, rant aside, so I booted up from the Ventoy and saw that I only had Endeavour OS anymore on it. Slapping myself upside the head, I decided to boot back into Manjaro and download the ISO again. And that’s when I first got that error message about being unable to mount /boot. As far as I can see, it’s some kind of an emergency mode in the initramfs — busybox perhaps. But it doesn’t let me do anything other than reboot by way of the three-finger salute.

Now, I still thought that I could fix whatever needed to be fixed once I had the conversion behind me, and so I went ahead and started the conversion anyway, by way of the Endeavour OS ISO and a manual chroot, all with man pages open and firefox aimed at several pages of the Arch Wiki.

I did however make certain that there were no errors on the filesystem itself — given that I had created a filesystem LABEL for it, which it did not have yet before — and in the process, I also checked the btrfs volume for errors, and I ran a full “btrfs scrub” on it too after copying all my files back — by hand, one major directory at the time! — into the individual subvolumes, from a mounted timeshift rsync backup on my spinning-rust HDD.

I have also examined the contents of the initramfs with lsinitcpio — I even unpacked it into the home directory of the live user of the Endeavour OS USB — so as to see what was all there, and most notably, my meticulously formatted new /etc/fstab, which I am including in the initramfs by way of the FILES= section of /etc/mkinitcpio.conf.

The reason why I am including /etc/fstab in there is because I now have a very specialized setup, with /etc being on its own subvolume in a flat list, instead of as a directory in the “@” subvolume mounted as /.

If I had made /etc into a nested subvolume under “@”, then its contents would already be visible from the moment “@” gets mounted, but it would then be mounted with the same mount options as “@” itself, and you can then explicitly mount it with its own mount options later.

Perhaps I should have done it that way — and I can always move it under “@” if I have to — but I wanted it to be a flat list, which looks nicer, because otherwise it would have been the only nested subvolume. It’s just my OCD — and unlike in the case of many people who casually say they have OCD, I do actually have OCD.

Anyway, below is my new layout — or at least, on account of /dev/sda, because /dev/sdb is my HDD and is only ever mounted while making backups. Caution: It’s a big list.

# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a device; this may
# be used with UUID= as a more robust way to name devices that works even if
# disks are added and removed. See fstab(5).
#
#
#############################################################################################################################################################
#
# FILESYSTEM UUID                           PARTITION TYPE              PARTITION LABEL      FILESYSTEM LABEL       FILESYSTEM FORMAT
#
#
# CEF6-EF5C                                 EFI System Partition        ESP                  ESP                    vfat
# 924f11d8-4bec-49a6-852e-033ce5e3d6a3      "/boot" filesystem          BOOT                 BOOT                   ext4
# 3f1852bc-be95-4e42-8d46-25bafac4ce79      btrfs root filesystem       SYSTEM               SYSTEM                 btrfs
#
#############################################################################################################################################################
#
# PARTITION LABEL  MOUNTPOINT          FSTYPE  MOUNT OPTIONS                                                                                       DUMP  PASS
#
PARTLABEL=ESP      /boot/efi           vfat    ro,defaults,noatime,sync,nodev,umask=0022,tz=UTC                                                    0     0
PARTLABEL=BOOT     /boot               ext4    ro,defaults,noatime,nodev                                                                           0     0
PARTLABEL=SYSTEM   /                   btrfs   ssd,ro,noatime,space_cache,compress=zstd,nodev,discard=async,subvolid=257,subvol=@                  0     0
PARTLABEL=SYSTEM   /usr                btrfs   ssd,ro,noatime,space_cache,compress=zstd,nodev,discard=async,subvolid=272,subvol=@usr               0     0
PARTLABEL=SYSTEM   /usr/local          btrfs   ssd,ro,noatime,space_cache,compress=zstd,nodev,discard=async,subvolid=273,subvol=@usr.local         0     0
PARTLABEL=SYSTEM   /opt                btrfs   ssd,ro,noatime,space_cache,compress=zstd,nodev,discard=async,subvolid=274,subvol=@opt               0     0
PARTLABEL=SYSTEM   /var                btrfs   ssd,rw,noatime,space_cache,compress=zstd,nodev,discard=async,subvolid=275,subvol=@var               0     0
PARTLABEL=SYSTEM   /var/cache          btrfs   ssd,rw,noatime,space_cache,compress=zstd,nodev,discard=async,subvolid=276,subvol=@var.cache         0     0
PARTLABEL=SYSTEM   /var/log            btrfs   ssd,rw,noatime,space_cache,compress=zstd,nodev,discard=async,subvolid=277,subvol=@var.log           0     0
PARTLABEL=SYSTEM   /home               btrfs   ssd,rw,noatime,space_cache,compress=zstd,nodev,discard=async,subvolid=278,subvol=@home              0     0
PARTLABEL=SYSTEM   /srv                btrfs   ssd,rw,noatime,space_cache,compress=zstd,nodev,discard=async,subvolid=279,subvol=@srv               0     0
PARTLABEL=SYSTEM   /srv/mmedia/music   btrfs   ssd,rw,noatime,space_cache,compress=zstd,nodev,discard=async,subvolid=280,subvol=@srv.mmedia.music  0     0
PARTLABEL=SYSTEM   /srv/mmedia/video   btrfs   ssd,rw,noatime,space_cache,compress=zstd,nodev,discard=async,subvolid=281,subvol=@srv.mmedia.video  0     0
PARTLABEL=SYSTEM   /var/spool          btrfs   ssd,rw,noatime,space_cache,compress=zstd,nodev,discard=async,subvolid=282,subvol=@var.spool         0     0
PARTLABEL=SYSTEM   /root               btrfs   ssd,rw,noatime,space_cache,compress=zstd,nodev,discard=async,subvolid=283,subvol=@rootuser          0     0
PARTLABEL=SYSTEM   /var/tmp            btrfs   ssd,rw,noatime,space_cache,compress=zstd,nodev,discard=async,subvolid=284,subvol=@var.tmp           0     0
PARTLABEL=SYSTEM   /etc                btrfs   ssd,ro,noatime,space_cache,compress=zstd,nodev,discard=async,subvolid=285,subvol=@etc               0     0

# Below is the root subvolume of a btrfs filesystem on the MDT HDD, which is used exclusively for backups.
UUID=1aedac7f-403b-4d85-9643-0d622be75453   /mnt       btrfs   noauto,nofail,subvolid=5,defaults,space_cache,autodefrag,compress=zstd,nodev,noexec 0     0

And just to be complete, here’s my /etc/mkinitcpio.conf…

# vim:set ft=sh
# MODULES
# The following modules are loaded before any boot hooks are
# run.  Advanced users may wish to specify all system modules
# in this array.  For instance:
#     MODULES=(usbhid xhci_hcd)
MODULES=()

# BINARIES
# This setting includes any additional binaries a given user may
# wish into the CPIO image.  This is run last, so it may be used to
# override the actual binaries included by a given hook
# BINARIES are dependency parsed, so you may safely ignore libraries
BINARIES=()

# FILES
# This setting is similar to BINARIES above, however, files are added
# as-is and are not parsed in any way.  This is useful for config files.
FILES=(/etc/fstab)

# HOOKS
# This is the most important setting in this file.  The HOOKS control the
# modules and scripts added to the image, and what happens at boot time.
# Order is important, and it is recommended that you do not change the
# order in which HOOKS are added.  Run 'mkinitcpio -H <hook name>' for
# help on a given hook.
# 'base' is _required_ unless you know precisely what you are doing.
# 'udev' is _required_ in order to automatically load modules
# 'filesystems' is _required_ unless you specify your fs modules in MODULES
# Examples:
##   This setup specifies all modules in the MODULES setting above.
##   No RAID, lvm2, or encrypted root is needed.
#    HOOKS=(base)
#
##   This setup will autodetect all modules for your system and should
##   work as a sane default
#    HOOKS=(base udev autodetect modconf block filesystems fsck)
#
##   This setup will generate a 'full' image which supports most systems.
##   No autodetection is done.
#    HOOKS=(base udev modconf block filesystems fsck)
#
##   This setup assembles a mdadm array with an encrypted root file system.
##   Note: See 'mkinitcpio -H mdadm_udev' for more information on RAID devices.
#    HOOKS=(base udev modconf keyboard keymap consolefont block mdadm_udev encrypt filesystems fsck)
#
##   This setup loads an lvm2 volume group.
#    HOOKS=(base udev modconf block lvm2 filesystems fsck)
#
##   This will create a systemd based initramfs which loads an encrypted root filesystem.
#    HOOKS=(base systemd autodetect modconf kms keyboard sd-vconsole sd-encrypt block filesystems fsck)
#
##   NOTE: If you have /usr on a separate partition, you MUST include the
#    usr and fsck hooks.
#
#    NOTE from Aragorn: The above is not needed with the systemd hook

HOOKS=(systemd autodetect microcode modconf kms keyboard sd-vconsole block filesystems)

# COMPRESSION
# Use this to compress the initramfs image. By default, zstd compression
# is used for Linux ≥ 5.9 and gzip compression is used for Linux < 5.9.
# Use 'cat' to create an uncompressed image.
#COMPRESSION="zstd"
#COMPRESSION="gzip"
#COMPRESSION="bzip2"
#COMPRESSION="lzma"
#COMPRESSION="xz"
#COMPRESSION="lzop"
#COMPRESSION="lz4"
COMPRESSION="zstd"

# COMPRESSION_OPTIONS
# Additional options for the compressor
#COMPRESSION_OPTIONS=()
# MODULES_DECOMPRESS
# Decompress loadable kernel modules and their firmware during initramfs
# creation. Switch (yes/no).
# Enable to allow further decreasing image size when using high compression
# (e.g. xz -9e or zstd --long --ultra -22) at the expense of increased RAM usage
# at early boot.
# Note that any compressed files will be placed in the uncompressed early CPIO
# to avoid double compression.
#MODULES_DECOMPRESS="no"

As I said, all of the filesystems check out fine — I’ve checked both the ext4 partition at /boot (with e2fsck) and I’ve checked the btrfs filesystem (with “btrfs check” and “btrfs scrub”). The hardware is also still in good condition – I’ve checked that too.

And as I also said earlier, the problem with the initramfs already occurred just before I went ahead with the conversion of my partitioning, with the only possible clue being that zstd error during the automatic rebuild of the initramfs after the removal of the mkinitcpio-openswap package.

So, here I am, running from that gawdawful Endeavour OS live session. I don’t often post a support request, and when I do, it’s usually about very specialized things that only a few people know about. But neither the documentation nor DuckDuckGo’s otherwise quite congenial duck.ai can offer me any solace.

I am genuinely at the end of my rope. I’ve been going at this for over 24 hours already, and I’m also battling some other technical issues here at my home in the meantime as well. So I’m hoping one (or several) of you people can help me out with this one.

I am obviously missing something. But the question is “What?”

Aragorn · 5 June 2025 02:53

Well, I think I found something related to the broken pipe, and that was indeed the case for me…

run0, mkinitcpio and zstd error / System Administration / Arch Linux Forums

I had indeed escalated my privileges with run0 instead of with “su -”, as I normally do.

Still, this still doesn’t explain the broken boot sequence, which I already experienced before I ever started the conversion described in my opening post.

soundofthunder · 5 June 2025 02:55

I have no idea what caused the problem, but I have a hunch that changing the compression method might bypass the error (of course, being a guess, it might not help at all).

This looks like it might be related; so, possibly an old bug may be behind this, or at least related:

zstd error 70 write error cannot write block broken pipe on all kernels from archinstall

Which is why I’m thinking the compression change might help.

#COMPRESSION="zstd"
#COMPRESSION="gzip"
#COMPRESSION="bzip2"
#COMPRESSION="lzma"
#COMPRESSION="xz"
#COMPRESSION="lzop"
#COMPRESSION="lz4"
COMPRESSION="zstd"

If you were using Nvidia I’d suggest this might be relevant:

[SOLVED] mkinitcpio: Write error : cannot write block : Broken pipe

Especially the links given in post #2 – but, you’re using Intel graphics.

Aragorn · 5 June 2025 03:01

Indeed, it’s an MSI motherboard with a 6-core Intel i5-8400 and an integrated Intel UHD 630.

soundofthunder · 5 June 2025 03:20

I saw mentioned in a few offsite articles that insufficient space for /boot can generate the same (or similar) error; and now, in the post you linked, Seth mentions space being a factor, again (with the kms hook).

Are you certain there’s (still) enough free on the /boot partition?

Aragorn · 5 June 2025 03:30

Yes, but that’s in the case of the Nvidia drivers, which does not apply to me. Furthermore, the link I provided higher up clearly states that it’s due to a bug in run0, which is what I was indeed using for elevating my privileges at that point in time.

Yes, I am. It’s only being used by about 16%. I only have one kernel in there, and no fallback initramfs.

linux-aarhus · 5 June 2025 04:33

Regarding your boot partition - when suspecting something has gone awry in the write of the label - I would remove the partition, recreated it with the proper label - then format the partition and copy the files back.

As for mkinitcpio-openswap - the package is included on all ISO - it is used when the system is encrypted and is needed when the swap is encrypted and hibernation is enabled.

Manjaro get-iso script found at Applications / Manjaro Get Iso · GitLab work anywhere suffice the python-requests package is available.

sudo pacman -U https://mirror.easyname.at/manjaro/pool/overlay/manjaro-get-iso-0.17.0-1-any.pkg.tar.zst

That would require two sticks because the Ventoy partition is unavailable on the booted stick.

Aragorn · 5 June 2025 04:37

That was indeed one of my thoughts as well, and I will keep this into consideration — thank you.

Ah, that does not apply to me then, because…

This is a desktop system, not a laptop, and thus there is no need for encryption.
I do not have or use a swap device.
I do not hibernate or suspend my system.

I had completely forgotten about that — thank you.

Actually, no, I can open it. Ventoy is not mounted anymore once the live session has finished booting.

(And I only have one stick of this capacity. My other ones are old and only have a 2 GiB capacity.)

linux-aarhus · 5 June 2025 04:45

Oh - I didn’t check it recently - that is an improvement - it has not always been so - in fact it is one of the reasons why ventoy provides the option of leaving empty space at a given size.

Feel free to use my private Manjaro (plasma) based rescue ISO at linux-aarhus archive @ uex.dk

I just checked - the Ventoy partition is not available

andreas85 · 5 June 2025 16:20

That is a rather complex setup

That’s a pretty complex setup you have there. But I’m in a similar situation. A lot of stuff accumulates over time. (Because Manjaro is so stable and doesn’t need to be reinstalled every two years.)

Let’s assume the file systems are OK (you’ve tested them).

What do you need to boot?

UEFI (needs an entry pointing to the correct partition)
GRUB (must be patched so it looks for its grub.cfg on the correct partition)
Kernels (must be in the right place and complete)
initramdisk (must be in the right place and complete)

You can easily get an overview with a program I wrote some time ago that lists the above information in a coherent manner. (It requires Java to be used, but as a single file, it’s not difficult to download and run.)

sudo wget https://github.com/andreaskielkopf/maxi/raw/master/jar/maxi -O /usr/local/bin/maxi ; sha256sum /usr/local/bin/maxi ; sudo chmod -c a+x /usr/local/bin/maxi

then

maxi -h

or

java -jar ./maxi -h

What you can try:

disable compression for initramdisk (cat)
use fallbackramdisk to boot
use another kernel
try to “ESC” grub and see if the grub.cfg is from the right partition
test if grub has the right modules (btrfs … ) loaded
reinstall grub (! because of zst that was not part of grub at some time in the past)
remove some complexity from your setup
copy /boot into @
copy /etc into @

has grub finished his job ?
or does grub search for its modules to load them ? (at /boot/grub/x86_64-efi )
maybe it does not find the /etc/fstab in the initramdisk
is /boot/grub … included in @ ?

Aragorn · 5 June 2025 19:11

Yes, that has not changed.

Nothing has changed there either.

All of that is in the right place, because those partitions are still the same ones as before.

The only thing that’s really different now is that the root partition (/dev/sda3) has been enlarged, and that everything else I had on separate partitions behind the root filesystem is now a subvolume on the root partition.

I thought about that, but I don’t think it’s really necessary yet, given that /boot is still ext4.

If it turns out — see farther down — that there’s a problem with that filesystem (even though it checked out okay), then I’ll be replacing it with btrfs, which so far appears to be far more robust than ext4, and in that case, I’ll disable compression on the initramfs due to btrfs having its own compression already.

I currently no longer have that — I’ve disabled the creation of a fallback in the kernel preset file.

However, I could re-enable it, if necessary. It’s an option.

I’d rather not. 6.12 LTS has served me very well so far.

It’s still where it used to be — in /boot, which is on /dev/sda2. But I have enabled more verbosity in /etc/default/grub now, so upon my next attempt, I should get a clearer view of what the problem could be.

It does, because my root partition has always been btrfs. All that has changed in that regard is that the root partition is now much bigger.

Again, nothing has changed there, but I’ve already reinstalled grub several times now.

Well, I’ll see what the kernel says upon my next attempt to boot. If necessary, I can indeed move the @etc subvolume under @ as a nested subvolume. Technically it shouldn’t be necessary because I’m already including /etc/fstab in the initramfs, but it’s an option I’m willing to consider.

Yes — the kernel is loaded, but it’s stuck in the kernel’s own internal initramfs, which does of course not contain all of the modules. It appears to fail at locating the on-disk initramfs, which contains all of the required modules and files.

I suspect that it has a problem mounting /boot – it also complains that it cannot mount /boot/efi, which is logical, since its mountpoint resides on the partition holding /boot.

Again, I’ll see what gives upon the next boot attempt.

That is possible, yes, and in that case I’ll have to make @etc a nested subvolume under @.

No, /boot is a separate partition — ext4 at the moment, due to grub not being able to write to btrfs.

soundofthunder · 6 June 2025 04:35

For what it’s worth, surely someone must have a copy of a manjaro-architect ISO that might still be accessible - the last copy I had weighed in at ~ 500MB.

I can’t recall exactly what features it had; perhaps you might still find a use for it.

linux-aarhus · 6 June 2025 04:36

Aragorn · 6 June 2025 05:40

Guys, I’m almost there. Several things have come up in this thread, and there were multiple causes to the problem, among which — as I feared already, and as @linux-aarhus also pointed out — a corruption of the superblocks in my ext4 partition at /dev/sda2.

I’m not sure what caused that — was it the zstd broken pipe running via elevated privileges through run0, or was it something else? — but I ended up reformatting that partition with btrfs.

But, when mounting it from the live session, it also threw up an error with a number of potential causes, among which “bad superblock”. So I began to suspect — whether correctly or not; the experiment explained farther down will either confirm or refute that — that the SSD had contracted damage, and I ended up making /boot a subvolume on the main btrfs partition at /dev/sda3, thereby leaving a 512 MiB gap of unused drive space between the EFI partition and my main system partition. I then renumbered the partitions in the partition table, so that /dev/sda3 is now /dev/sda2.

But as I’ve just discovered, even though the superblock was damaged on the ext4 filesystem that was there earlier, you get the exact same error message at mount time if you specify the wrong space_cache in the btrfs mount options.

The “space_cache” mount option (without options) assumes version 1, but newer kernels all default to version 2, which must then either not be explicitly listed as a mount option, or explicitly listed as “space_cache=v2”. And considering that I had only enlarged the btrfs filesystem that was already on /dev/sda3 from back in 2019, it still had version 1 of the space_cache, while the newly created partition was version 2 already.

Now, I am currently already running off of my installed system — and as if it all wasn’t bad enough already, I also had an additional problem with DNS at boot time, thanks to a crash of systemd-resolved (which I have now disabled in favor of NetworkManager’s own name resolution) — but considering that my root filesystem is still using space_cache version 1, I’m going to convert that to version 2, which I cannot do from a running system, because the filesystem must be unmounted.

In other words, I’m going to have to boot up from that live USB again — Endeavour OS, yuk, but it’ll do for now — and in the process, I’m going to try and see whether I can recreate the original /boot partition (as btrfs) in that 512 MiB gap between the EFI partition and the main partition, but without deleting the /boot subvolume on the main partition — I’ll just comment it out in /etc/fstab and add a new line for /dev/sda2. If it works, then I can delete the @boot subvolume, and if it doesn’t, then I can easily return to a booting system.

I’m not sure whether the @etc subvolume being under the top level was one of the reasons for the failure to mount /boot — I suspect it might have been, even though the boot-up messages showed that my /etc/fstab had indeed been loaded into the initramfs by mkinitcpio, and that everything was being mounted from within there — so I’ve moved it under @ and renamed it to simply /etc. It’s still a separate subvolume, but now it — and therefore, /etc/fstab — can already be found by the initramfs before it gets explicitly mounted as a separate subvolume.

I’ll also have to change the partition order again after recreating the separate /boot partition, because right now, what used to be /dev/sda3 is actually /dev/sda2.

I’ll report back as soon as everything’s set up properly.

Aragorn · 6 June 2025 12:59

I’m back!

It… didn’t quite go as I had anticipated… Aging eyes and a lot of stress and anxiety, and the result was an incredibly stupid typo that ruined my whole installation. So I ended up having to redo the whole thing and restore my system from a freshly made backup. I’ve only just finished now — rebooted into my installed system at exactly 14:10 Central European Time (12:10 UTC).

But, there is also good news…

My SSD is not damaged, because the problem with the superblock was only in that ext4 partition, and the failure to mount the partition again once it had been reformatted to btrfs was the result of the wrong space_cache format.
I have created a new btrfs /boot partition at /dev/sda2 — /dev/sda1 is my EFI partition and /dev/sda3 is the main system partition with the btrfs subvolumes. And it’s all space_cache version 2, baby!
I now also have a whole lot more free space than before, and the free space is shared among all subvolumes. With fixed partitions, you have to pay attention, and if they have a btrfs filesystem on them, then you may also regularly need to balance them in order to free up extra space — as I had to do with my old /usr partition after each bundled update. But no more of that now.
As harrowing as this whole experience was, I have now learned a lot more about filesystems and partitioning than I already knew.

Anyway, now that I’m a happy camper again — he says, looking around for any swords hovering over his neck — I would like to thank the three people who’ve helped me with clues…:

But ultimately, it was a combination of things. At first it was the bad superblock on the ext4 partition, which I already suspected, and which @linux-aarhus then also posited as a possibility, and then later it was the space_cache version issue, which I myself discovered. And then I royally screwed up and ruined my installation, and I had to start all over again.

So in the end, I’m going to claim the solution for myself. I will however leave you all with this…

Don’t try this at home, kids!

andreas85 · 6 June 2025 15:29

(Willkommen im Club)

Welcome to the club.

The only thing you’re missing now is btrfs RAID (and then backsnap)

Yochanan · 6 June 2025 19:28

Well, you did.

Aragorn · 9 June 2025 19:28

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.