Errors on boot after a failed update; using btrfs snapshots

Background: The other night, I was trying to watch a video and X locked up. I had a pacman -Syu running at the time so I decided to just leave the computer on overnight. The next morning I rebooted (I was able to ssh in from another computer; I just ran sudo reboot from the command line) but was unable to boot; after I booted from a Manjaro live environment, I found that the kernel files (vmlinuz-*) were missing from my /boot partition.

I was able to do a manjaro-chroot and reinstall kernel 4.19. But whatever grub update ran after that seems to have messed up my boot, because now I just get a blinking cursor when I attempt to boot that drive.

I'm able to boot using the Manjaro USB stick and the grub command line.

set root='hd4,gpt1'
linux /vmlinuz-4.19-x86_64 root=/dev/sda3 rw rootflags=subvol=@/.snapshots/29/snapshot resume=/dev/sda2
initrd  /intel-ucode.img /initramfs-4.19-x86_64.img

(where sda1 = /boot, sda2 = swap, sda3 = /, sda4 = /home)

From there I can get to a mostly functional command line. No X, and I can only log in as root. My typical user ID is not in /etc/passwd; this is a very old snapshot, my most recent one is 4672.

(Another note: I am using snapper for snapshots. I recently used snapper delete to delete a number of old snapshots -- everything after 29 and before 3938. I don't know if this is relevant to my current problem; my understanding is that deleting old snapshots shouldn't impact the functionality of newer ones.)

From that snapshot 29 boot, I'm able to access the network and run pacman and mhwd-kernel. I wasn't able to reinstall kernel 5.3, the one I was using before; it gives me the error

linux53: /boot/linux53-x86_64.kver exists in filesystem

However, I was able to install kernels 5.4 and 5.5, since I hadn't installed them on this system before. I'm able to boot with those kernels, with the same behavior as 4.19.

Here's some diagnostic output from when I boot to snapshot 29:

inxi -Fxz

When I attempt to run any other snapshot later than 29, I get a series of errors, starting with

Failed to start Load Kernel Modules

and get dumped to a "Give root password for maintenace" prompt. I'm unable to load kernel modules, mount anything besides / (which is mounted read-only), or access the network, so I'm not sure how to grab output from dmesg.

So, to summarize:

I want to be able to boot to a later snapshot
I want to fix Grub so I don't have to use command-line grub to boot.

Can anyone tell me what my next step should be? Thanks.

No luck yet.

I'm not sure if this is relevant, but I do get some errors when I run btrfs check:

[manjaro@manjaro ~]$ sudo btrfs check /dev/sda3
Opening filesystem to check...
Checking filesystem on /dev/sda3
UUID: d19c87c1-8b50-46b8-84a3-bc640c0d79b8
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
block group 33285996544 has wrong amount of free space, free space cache has 157319168 block group has 157335552
failed to load free space cache for block group 33285996544
[4/7] checking fs roots
root 297 inode 234412 errors 1040, bad file extent, some csum missing
ERROR: errors found in fs roots
found 34874093570 bytes used, error(s) found
total csum bytes: 30348212
total tree bytes: 1181155328
total fs tree bytes: 1065992192
total extent tree bytes: 70811648
btree space waste bytes: 207567505
file data blocks allocated: 175085117440
 referenced 68374085632

Any suggestions? Anything else I should try?

My /home is on a separate partition and is in good functioning order. If I'm not able to figure out how to fix my root partition, I may reinstall / and /boot and keep my current /home. Any suggestions for things I should keep in mind if I decide to go that route?


well you should stop using snapper and use timeshift instead.but its my opinion.
you should also use grub-vanilla and grub-btrfs.
you should chroot and do a

sudo pacman -Syyuu 

Thanks, I'll look into grub-vanilla and grub-btrfs. I've used timeshift before on Mint; I can look into whether it's a better option than Snapper. Snapper sure didn't do me any good this time.

As of right now I've finally got my system back up and running again. I realized that if snapshot 29 was the only snapshot that sort-of worked, I should focus on rolling back to that one instead of trying to fix the later ones.

I could only log into snapshot 29 as root, because my user account wasn't in /etc/passwd. I'm not sure why (maybe it got overwritten when I was updating in a chroot?), but I realized I could just grab the correct /etc/passwd from one of the other snapshots. Once I did that, I could log in under my user account, and once I switched out my graphics drivers, I was able to get into X. (I bought a new graphics card in November; snapshot 29 was from June, so I had to go through mhwd -r pci video-nvidia-430xx and mhwd -i pci video-linux again.)

My Grub is still messed up, and I'm still having to boot from a grub> prompt from USB. I'll look into those Grub packages you recommended and let you know if they work.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

Forum kindly sponsored by