Btrfs out of unallocated and metadata space due to high fragmentation + heavy use > broke balancing + read-only system

It’s a bit like switching from Winbose to Linux: one knows the second is superior and better, but at the beginning it feels overwhelmingly complicated. One needs a bit of time and learning, and at some point after getting sufficient knowledge about it, it does not look so scary anymore.

5 Likes

That is a really good conclusion. For the meantime, I’m staying with btrfs, the raid options, recovery, modularity … are really impressive.

I’ve done the scrub today, once from live media, once from booted system on it’s own boot partition, didn’t accomplish much, but at least there wont be bitrot. I’ll see if manual checking is viable, if not, I’ll give btrfsmaintenance a try.. One warning, btrfs filesystem usage without elevation doesn’t show the Unallocated Volume, the possibly most important one.

I’ll try the long re-balancing tomorrow. Here are today’s numbers:



sudo btrfs filesystem usage /media/mint/9fa777d4-6ca8-4457-a1eb-eba91ac8eccd 
Overall:
    Device size:		 944.77GiB
    Device allocated:		 837.07GiB
    Device unallocated:		 107.70GiB
    Device missing:		     0.00B
    Used:			 782.24GiB
    Free (estimated):		 158.71GiB	(min: 104.86GiB)
    Data ratio:			      1.00
    Metadata ratio:		      2.00
    Global reserve:		 512.00MiB	(used: 0.00B)

Data,single: Size:827.01GiB, Used:776.00GiB (93.83%)
   /dev/mapper/luks-fcd41acc-db81-4f2d-9e84-159b7c178fdf	 827.01GiB

Metadata,DUP: Size:5.00GiB, Used:3.12GiB (62.43%)
   /dev/mapper/luks-fcd41acc-db81-4f2d-9e84-159b7c178fdf	  10.00GiB

System,DUP: Size:32.00MiB, Used:128.00KiB (0.39%)
   /dev/mapper/luks-fcd41acc-db81-4f2d-9e84-159b7c178fdf	  64.00MiB

Unallocated:
   /dev/mapper/luks-fcd41acc-db81-4f2d-9e84-159b7c178fdf	 107.70GiB
$ sudo btrfs scrub start -Bd /media/mint/9fa777d4-6ca8-4457-a1eb-eba91ac8eccd 
scrub device /dev/mapper/luks-fcd41acc-db81-4f2d-9e84-159b7c178fdf (id 1) done
Scrub started:    Tue May  5 12:22:58 2026
Status:           finished
Duration:         0:10:34
Total to scrub:   837.07GiB
Rate:             1.23GiB/s
Error summary:    no errors found
$ sudo btrfs filesystem usage /media/mint/9fa777d4-6ca8-4457-a1eb-eba91ac8eccd 
Overall:
    Device size:		 944.77GiB
    Device allocated:		 837.07GiB
    Device unallocated:		 107.70GiB
    Device missing:		     0.00B
    Used:			 782.24GiB
    Free (estimated):		 158.71GiB	(min: 104.86GiB)
    Data ratio:			      1.00
    Metadata ratio:		      2.00
    Global reserve:		 512.00MiB	(used: 0.00B)

Data,single: Size:827.01GiB, Used:776.00GiB (93.83%)
   /dev/mapper/luks-fcd41acc-db81-4f2d-9e84-159b7c178fdf	 827.01GiB

Metadata,DUP: Size:5.00GiB, Used:3.12GiB (62.43%)
   /dev/mapper/luks-fcd41acc-db81-4f2d-9e84-159b7c178fdf	  10.00GiB

System,DUP: Size:32.00MiB, Used:128.00KiB (0.39%)
   /dev/mapper/luks-fcd41acc-db81-4f2d-9e84-159b7c178fdf	  64.00MiB

Unallocated:
   /dev/mapper/luks-fcd41acc-db81-4f2d-9e84-159b7c178fdf	 107.70GiB

   
   
 sudo btrfs filesystem usage  /                                                                                                                                                                                                                                                                                                                                                                                         
Overall:
    Device size:                 944.77GiB
    Device allocated:            837.07GiB
    Device unallocated:          107.70GiB
    Device missing:                  0.00B
    Device slack:                    0.00B
    Used:                        782.26GiB
    Free (estimated):            158.69GiB      (min: 104.84GiB)
    Free (statfs, df):           158.69GiB
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:              512.00MiB      (used: 112.00KiB)
    Multiple profiles:                  no

Data,single: Size:827.01GiB, Used:776.01GiB (93.83%)
   /dev/mapper/luks-fcd41acc-db81-4f2d-9e84-159b7c178fdf         827.01GiB

Metadata,DUP: Size:5.00GiB, Used:3.12GiB (62.45%)
   /dev/mapper/luks-fcd41acc-db81-4f2d-9e84-159b7c178fdf          10.00GiB

System,DUP: Size:32.00MiB, Used:128.00KiB (0.39%)
   /dev/mapper/luks-fcd41acc-db81-4f2d-9e84-159b7c178fdf          64.00MiB

Unallocated:
   /dev/mapper/luks-fcd41acc-db81-4f2d-9e84-159b7c178fdf         107.70GiB

 sudo btrfs scrub start -Bd /                                                                                                                                                                                                                                                                                                                                                                                   
Starting scrub on devid 1

Scrub device /dev/mapper/luks-fcd41acc-db81-4f2d-9e84-159b7c178fdf (id 1) done
Scrub started:    Tue May  5 15:00:47 2026
Status:           finished
Duration:         0:11:03
Total to scrub:   781.64GiB
Rate:             1.18GiB/s
Error summary:    no errors found
 
 sudo btrfs filesystem usage  /                                                                                                                                                                                                                                                                                                                                                                               
Please enter the PIN: 
Please touch the FIDO authenticator.
Overall:
    Device size:                 944.77GiB
    Device allocated:            837.13GiB
    Device unallocated:          107.63GiB
    Device missing:                  0.00B
    Device slack:                    0.00B
    Used:                        781.59GiB
    Free (estimated):            159.27GiB      (min: 105.46GiB)
    Free (statfs, df):           159.27GiB
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:              512.00MiB      (used: 0.00B)
    Multiple profiles:                  no

Data,single: Size:827.01GiB, Used:775.37GiB (93.76%)
   /dev/mapper/luks-fcd41acc-db81-4f2d-9e84-159b7c178fdf         827.01GiB

Metadata,DUP: Size:5.00GiB, Used:3.11GiB (62.24%)
   /dev/mapper/luks-fcd41acc-db81-4f2d-9e84-159b7c178fdf          10.00GiB

System,DUP: Size:64.00MiB, Used:128.00KiB (0.20%)
   /dev/mapper/luks-fcd41acc-db81-4f2d-9e84-159b7c178fdf         128.00MiB

Unallocated:
   /dev/mapper/luks-fcd41acc-db81-4f2d-9e84-159b7c178fdf         107.63GiB


So I’ve run the aggressive balance as well, suprisingly, it only took 12 minutes and got me 20 extra gigs of unallocated.

   sudo btrfs filesystem usage  /
Overall:
    Device size:                 944.77GiB
    Device allocated:            840.07GiB
    Device unallocated:          104.70GiB
    Device missing:                  0.00B
    Device slack:                    0.00B
    Used:                        789.22GiB
    Free (estimated):            151.75GiB      (min: 99.40GiB)
    Free (statfs, df):           151.75GiB
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:              512.00MiB      (used: 0.00B)
    Multiple profiles:                  no

Data,single: Size:830.01GiB, Used:782.95GiB (94.33%)
   /dev/mapper/luks-fcd41acc-db81-4f2d-9e84-159b7c178fdf         830.01GiB

Metadata,DUP: Size:5.00GiB, Used:3.14GiB (62.72%)
   /dev/mapper/luks-fcd41acc-db81-4f2d-9e84-159b7c178fdf          10.00GiB

System,DUP: Size:32.00MiB, Used:128.00KiB (0.39%)
   /dev/mapper/luks-fcd41acc-db81-4f2d-9e84-159b7c178fdf          64.00MiB

Unallocated:
   /dev/mapper/luks-fcd41acc-db81-4f2d-9e84-159b7c178fdf         104.70GiB
 sudo btrfs balance start -dusage=75 -musage=75 /
Done, had to relocate 32 out of 837 chunks
 sudo btrfs filesystem usage  /                                                                                                                                                        
Overall:
    Device size:                 944.77GiB
    Device allocated:            820.07GiB
    Device unallocated:          124.70GiB
    Device missing:                  0.00B
    Device slack:                    0.00B
    Used:                        789.37GiB
    Free (estimated):            153.61GiB      (min: 91.26GiB)
    Free (statfs, df):           153.61GiB
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:              512.00MiB      (used: 0.00B)
    Multiple profiles:                  no

Data,single: Size:812.01GiB, Used:783.09GiB (96.44%)
   /dev/mapper/luks-fcd41acc-db81-4f2d-9e84-159b7c178fdf         812.01GiB

Metadata,DUP: Size:4.00GiB, Used:3.14GiB (78.43%)
   /dev/mapper/luks-fcd41acc-db81-4f2d-9e84-159b7c178fdf           8.00GiB

System,DUP: Size:32.00MiB, Used:128.00KiB (0.39%)
   /dev/mapper/luks-fcd41acc-db81-4f2d-9e84-159b7c178fdf          64.00MiB

Unallocated:
   /dev/mapper/luks-fcd41acc-db81-4f2d-9e84-159b7c178fdf         124.70GiB

I think that my balance with the filesystem stretched over the USB drive probably did most of the balancing, so that’s why this task was possibly lighter.

I’ll definitely be doing the scrubs monthly and probably weekly balances, just much less aggressive, I’m thinking having 25 or 50% empty chunks moved.

What I’ve realized, had the error happened during the system update, I’ll be in much worse state, ofc, I’d still be able to eventually use a snapshot, but it would have been much harder. I’ll definitely be checking the unallocated size and fragmentation before each larger update now :slight_smile: .

Once again, thank you for all the advice :+1: .

Mod edit: Consecutive posts merged.

3 Likes

Glad it all worked out. Even though you were literally converting raid types, and adding unreliable flash sticks into the mix! Btrfs just kept everything in chunks, and you just moved them around. Nothing damaged whatsoever.

This is one of the great things about btrfs. Add or remove devices, convert raid profiles, rebalance data, all on your live root filesystem. And you even went the “wrong” direction for a while, and all was still fine in the end.

Just wait until you find out all the things you can do with these snapshots. :grinning_face:

Yes. :wink:

Weekly sounds like a bit much, and aggressive balances are rarely needed. There’s a point where all you are doing is extra SSD wear and IO. Especially with high snapshot retention and turn over. Certain workloads can accelerate allocator fragmentation and metadata pressure.

You generally do not want heavily-used databases, VM images, containers, or virtual disks living on heavily snapshotted CoW storage.

This is where I put certain things on xfs (or ext4 is fine too). But I default to everything on btrfs.

Snapshot cleanup/deletion is usually what fragments chunk allocation the most. So that is a good time to check it.

You can see that the balance reclaimed about 15 GiB of chunk allocation back into unallocated space, but overall the filesystem was already in fairly decent shape after the earlier recovery work.

Normally it is not worth running balances this aggressively unless there is a specific reason.

But now you know what to watch for, and honestly that is the important part.

And yes, 70-80% allocation/utilization is completely fine for Btrfs.

Still, it’s always good to experiment and learn. :slight_smile:

1 Like

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.