Could not boot due to brfs no-space condition. How should I keep using BTRFS?

Support
, , ,
1

Hi gurus,

I’m relatively new to Linux so please excuse the lack of my knowledge. I switched from Mint to Manjaro KDE about 2 months ago and I decided to use BTRFS because I liked the idea of selectable snapshots from GRUB. Unfortunately, I did not do enough of research and I hit a wall today as my BTRFS partition seemingly ran out of space and Manjaro could not boot up. I managed to recover it after 5 hours of struggling . It has been a good learning experience, but at a very bad time. You can read all about it bellow - maybe it will help someone in the future.

I would like to ask for your opinions on my bellow questions:

  1. How can I prevent btrfs from running out of space in the future? I is enough just to use cron and periodically perform btrfs balance?
  2. Why is Manjaro not performing automatic btrfs balancing or other maintenance to automatically free up space? What’s the recommended approach for long term stability?
  3. Is is possible my filesystem was somehow corrupted due to the bug reported here ?
  4. Is there any way to convert btrfs into ext4 when Manjaro was installed fresh with btrfs?

My story:

  • I have been experiencing high CPU utilization due to ‘kworker events_unbound’ . Based on this forum topic, this is known kernel bug. I suspect it was related to btrfs snapshot creation…
  • I upgraded from kernel 5.13.19-2 to 5.15.2-2 and I did not see this issue after reboot. I was happy, but not for long.
  • I tried to create a new snapshot in Timeshift , but I noticed that all past snapshots only showed a few bytes in size, which shocked me.
  • Few minutes later, most of my apps including plasma-shell crashed and I could not recover from it, so I rebooted.
  • System never booted up and I could only use shell after pressing (alt+f2).
  • I could not perform almost any operation due to ‘No Disk Space’ or ‘read-only’ errors.
    -Journalctl showed the bellow interesting line:
kernel: BTRFS warning (device nvme0n1p3): btrfs_uuid_scan_kthread failed -28

-I seemed to have enough of free disk space:

sudo btrfs fi df
Data, single: total=198.67GiB, used=169.28GiB
System, single: total=4.00MiB, used=48.00KiB
Metadata, single: total=4.01GiB, used=3.71GiB
GlobalReserve, single: total=315.77MiB, used=2.91MiB


sudo btrfs fi usage /
Overall:
   Device size:              202.68GiB
   Device allocated:		 202.68GiB
   Device unallocated:		 1.00MiB
   Device missing:		     0.00B
   Used:			         172.99GiB
   Free (estimated):		  29.39GiB	(min: 29.39GiB)
   Free (statfs, df):		  0.00B
   Data ratio:			      1.00
   Metadata ratio:		      1.00
   Global reserve:		      315.77MiB	(used: 3.83MiB)
   Multiple profiles:		  no
  • I was not able to delete or even access snapshots due to ‘No Disk Space’ error .
  • I managed to remove several large .iso files but it did not reflect anywhere.
  • I followed many articles including this, but nothing worked.
  • The only solution that I found to work was adding new USB disk to btrfs
  • I was able to access the partition now, but GRUB then failed to load. I assumed this was because both btrfs member disks showed with the same UUID and fstab needed some adjustment.
  • The last step to the solution was to run sudo btrfs balance start /<mount_point> and then removing the USB disk from btrfs.
  • At last I managed to boot into Manjaro, but all my snapshots were lost…
  • btrfs usage after balancing was completed:
sudo btrfs fi df /    
Data, single: total=130.00GiB, used=129.12GiB
System, single: total=32.00MiB, used=16.00KiB
Metadata, single: total=2.00GiB, used=1.18GiB
GlobalReserve, single: total=226.12MiB, used=0.00B

sudo btrfs fi usage /                                                                                                                                                                            
Overall:
    Device size:                 202.68GiB
    Device allocated:            132.03GiB
    Device unallocated:           70.65GiB
    Device missing:                  0.00B
    Used:                        130.30GiB
    Free (estimated):             71.54GiB      (min: 71.54GiB)
    Free (statfs, df):            71.53GiB
    Data ratio:                       1.00
    Metadata ratio:                   1.00
    Global reserve:              226.12MiB      (used: 0.00B)
    Multiple profiles:                  no
2

You should periodically running btrfs balance.

You could use a cron but is probably makes more sense to use the included systemd service/timer.

I don’t know if there is an official recommendation but I would recommend running btrfs balance and btrfs scrub. You might also look at the AUR package btrfsmaintenance which can make these tasks easier.

Are you running the specific kernel version that was impacted by that? If not, then no, not likely.

There is not an easy way to convert, no.

2 Likes
3

If you could recover, let alone interact with the system during breakage, it is by definition not ‘bricked’.

4

The laptop was about as useful as a brick to me for half a day. Also the system could not be interacted with - had to do it from other distro.

Thanks for your very valuable input anyway, I changed the subject.

5

Thanks, dalto , the package btrfsmaintenance was just the pointer I needed.

Shame the FS maintenance is not scheduled automatically on Manjaro. I read other distros do it by default, though everyone in forums seems to have different opinions on how frequently and to what degree to perform the balancing.

1 Like
6

I’ve been using BTRFS for over two years and I don’t use balancing. You just need to be aware that BTRFS works differently than other file systems. When you keep updating and creating snapshots they are using more and more space because of the diffrences between files. So, you need to keep track of how much snapshots you make and how much data changes between backups. After every Manjaro update you will see that disk space used is bigger that given in Manjaro update information. That is normal. And after creating backups the space frees.

google for BTRFS +disk space
for example

You have plenty of information about BTRFS here

2 Likes
7

I do use btrfs since years(at least 2015/16). I had to do this once. I did not lose any snapshot or any data.

The right way to “escape from jail”, is to:

  • add a partition (for example a extern USB-Stick with 8GB) to the btrfs-volume with btrfs devive add
  • From now on this usb-partition belongs to your volume! NEVER boot without it. I suggest to not reboot at all in this stage! This stick has to stay until btrfs device remove has completed.
  • now delete some stuff you do not need until your disk will be not more then 90% full
  • if there are old snapshots you don´t need, delete them now

Do not get confused if btrfs does not immediately display the vacant space. After the next step (balance) it becomes visible.

  • balance btrfs with a filter(!)
  • remove the added partition from the volume with btrfs device remove This will take some time ! Do not reboot until this step is complete.
  • Now you may remove the USB-Stick.

Now you are out of jail. Stay out !

  • Think about how to extend the btrfs volume to double size
  • seek if there is a program filling your disk
  • look for snapshots !
  • don’t forget to remove old snapshots (best done automatically)

please have a look at Btrfs - Manjaro

Balance will not prevent “out of space” !

https://wiki.manjaro.org/index.php/Btrfs#balance
This will only wear out your hard drive/ssd. Balance reads and writes every byte of your hard drive! It can shrink some space, but this is the wrong approach. If btrfs has enough space, an automatic defragmentation takes place. If there is not enough space, this can no longer work.

The right approach is:

When creating a btrfs volume, don’t be stingy on storage space. A btrfs volume should normally only be 80% full (mine is 46% with all snapshots included). Otherwise it is advisable to adjust the volume. In an emergency, 90% are ok too. But that’s not beneficial neither for the Btrfs volume nor for an SSD.

Your problem was:

Not the space used is the problem, but the space allocated. There is no space to breath for btrfs any more. And the disk is 85% Full. Dont´t do this with btrfs.

4 Likes
8

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.