Unable to boot after running out of disk space during an update

Hello,

I have a Dell XPS 13 9360 with Manjaro stable and the 5.15 kernel and Plasma. I am using btrfs.

I decided to perform an update today, which also included some AUR packages. The AUR packages include ceph-llibs, deadbeef and some others. ceph in particular required a lot of time and resources to build and has failed a few times before. During the update, it seems that I ran out of disk space based on the messages in pamac, and the desktop environment seemed broken (the KDE tray stopped appearing) and I wasn’t sure what to do…

When I checked the disk space with df -h, it seemed like I had about 13GB left, which I thought might be due to a cleanup, or maybe it’s due to btrfs not being able to calculate free space very well (from what I heard)

I decided to reboot the machine using the shutdown -r command, but that was stuck, so I force switched it off.

Now when I switch it on, it seems to get stuck while booting. this is what I get:

I’m not sure what to do next. I tried to get into grub to enable verbose logging, but I’m not sure how to do that either because it never appears after I power on the computer, and I’m using UEFI.

Hi @9a3eedi,

I’d recommend you boot from a live USB Thumb drive, enter a chroot environment, and continue/restart/redo the update from there.

2 Likes

It’s possible you have too many snapshots taking space. Do as @Mirdarthos suggests, but also check for the amount of space taken with snapshots. You can deal with it using the Timeshift app from the Manjaro’s Live Iso you created. You can try to restore to a previous point as well.

I believe you can safely remove it, the package has been dropped from the repository to aur.

1 Like

You will also have to do something about your lack of disk space and it will become a matter of regular maintenance to prevent such problems in the future.

I’ve attempted to do this, however, I’m not sure how… btrfs has some subvolumes and I am somewhat unfamiliar with how it works. I mounted the @ subvolume into /mnt, and chroot worked… but I’m not sure if that’s the correct method or if it’s enough. pacman -Syyu didn’t workbecause it couldn’t open /etc/mtab.

I’ve also tried to use manjaro-chroot -a /mnt but it didn’t work, it could not detect a Linux parititon. Timeshift on the LiveUSB also didn’t seem to detect my Manjaro installation.

In the meantime I’m looking for things to delete on my btrfs partition… but what’s interesting is that df -h still reports that I have 12GB of space left (2%)…

I think I was simply caught off guard this time, didn’t not realize I had low disk space and I went for an upgrade.

I apologize. I didn’t realize or I missed it in your post or forgot by the time I replied. I know absolutely nothing regarding BTRFS, so I’m out, here.

Hope you manage, though!

I also do not use btrfs, but the following may be helpful for you?

Keep Calm :wink:

The “no space left” condition can be handled. I have done this several times now. You can do it too.

There is a special section about “no space left”

If you need help, or are unsure:

You need:

  1. an actual (not older then 12 month) live usb or live iso (e.g. with kernel 5.15)
  2. a spare partition or usb-stick with at least 10GB size
  3. The partition needs to have NO filesystem present

What to do:

1. boot into live manjaro

2. No chroot necessary, but you need to become root

sudo su -

3. make dir /mnt/BTRFS in the live system

mkdir /mnt/BTRFS

4. mount the btrfs-partition with the “no space left” problem like:

(dont forget to replace /dev/sdc9 with your “no space left”-partition

mount -t btrfs -o subvol=/,compress=zstd:8 /dev/sdc9 /mnt/BTRFS

5. have a look (and post it)

ls -lA /mnt/BTRFS

6. have a look at your btrfs filesystem (and post it)

Look for Unallocated:

btrfs filesystem usage /mnt/BTRFS

7. prepare your spare partition

…

8. add it to the “no space volume”

btrfs device add /dev/sde99 /mnt/BTRFS

9. delete some files

…

10. balance

These balances may take a long time! (e.g. several hours) because the complete volume may be rewritten in this process.

first do

 btrfs balance start -musage=50 -dusage=50 /mnt/BTRFS

later then

 btrfs balance start -musage=90 -dusage=90 /mnt/BTRFS

and then

 btrfs balance start -musage=95 -dusage=95 /mnt/BTRFS

11. have a look at your btrfs filesystem (and post it)

Look for Unallocated:

btrfs filesystem usage /mnt/BTRFS

when there is enough unallocated space to remove the spare partition, and additionaly at least 10GB, then

12. remove(==delete) the spare partition from /mnt/BTRFS

 btrfs device remove /dev/sde99 /mnt/BTRFS     

Try reading and understanding the manjaro-wiki about btrfs

After that’s done. You should complete the interrupted update :wink:

Andreas :footprints:

1 Like

df -h will not warn you when btrfs gets out of space :frowning:

You have to use:

sudo btrfs filesystem usage /

:warning: And you have to look for Unallocated:

1 Like

I’ve actually solved the problem already but forgot to post about it. I followed similar steps to yours. Thank you very much for such a clear post!

This is what I did from what I can remember:

Make a LiveUSB of manjaro and boot into it. Mount the “@” subvolume of the btrfs partition
mount -o subvol=@ /dev/sda1 /mnt

Enter the pamac temp build directory and delete the ceph-libs build logs (which took around 2GB of space). It was somewhere in /var/tmp/pamac or /var/tmp/pamac-build, can’t remember

I also deleted some files in my home directory by mounting it first.
mount -o subvol=@home /dev/sda1 /mnt/home

Enter a chroot
manjaro-chroot /mnt

Do a pamac update
pamac update

Then, exit the chroot and reboot.

After rebooting, i checked timeshift and deleted some older snapshots and also looked for large files to delete and deleted those.

This was enough to fix the problem :slight_smile: Lesson learned: I should be more careful about running out of space on btrfs. However it seems really bad that common tools cannot report how much space is left in the btrfs filesystem.

1 Like

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.