Btrfs out of unallocated and metadata space due to high fragmentation + heavy use > broke balancing + read-only system

Support System Updates
1

Link to update

Yesterday, around noon, I have updated to 26.1 from 26.0.4, using “pacman -Syyu”
I went from kernel 6.6.128 to 6.6.137
The boot disk is formatted by Manjaro as btrfs with LUKS. No other OS present, grub is the bootloader.
Everything went fine, I’ve rebooted, system ran fine.
In the evening, around 4 hours of uptime, plasma crashed (no DE UI, GUI windows still open).
I restored it with systemctl --user restart plasma-plasmashell.service
Got another hour of uptime, when it started acting up again (I don’t remember what, just decided to restart).

Since then, it never booted into my DE again, TTY works, I can switch to another snapshot from grub, but neither the oldest (some older 26.0.4) nor the newest or any in between work.

I’ve tried to debug from TTY and when booting from USB.

Issue seems to be with btrfs, I can write to no part of the disk that has btrfs. When booting from USB, I’ve tried to make ext4 partition on the last 5MB of unallocated space of the disk, and I can write to that just fine. So basically, the main OS partition is now read-only filesystem, which is something plasma can’t boot into.

I’ve tried the various btrfs utilities, please understand, since I do debugging from TTY and liveboot OS, getting output out isn’t always easiest. If you need specific output and if I still have the system, let me know, this is what I’ve run and tried:

Basically, I’ve tried the repair tools and looked at the volumes and btrfs subvolumes.


$ sudo btrfs check --readonly /dev/mapper/luks-fcd41acc-db81-4f2d-9e84-159b7c178fdf
Opening filesystem to check...
Checking filesystem on /dev/mapper/luks-fcd41acc-db81-4f2d-9e84-159b7c178fdf
UUID: 9fa777d4-6ca8-4457-a1eb-eba91ac8eccd
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space tree
cache and super generation don't match, space cache will be invalidated
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
found 901755981824 bytes used, no error found
total csum bytes: 874434712
total tree bytes: 4832034816
total fs tree bytes: 3677208576
total extent tree bytes: 146440192
btree space waste bytes: 939024736
file data blocks allocated: 3019697774592
 referenced 2245901139968
$ sudo btrfs check --clear-space-cache v2 /dev/mapper/luks-fcd41acc-db81-4f2d-9e84-159b7c178fdf
Opening filesystem to check...
Checking filesystem on /dev/mapper/luks-fcd41acc-db81-4f2d-9e84-159b7c178fdf
UUID: 9fa777d4-6ca8-4457-a1eb-eba91ac8eccd
Clear free space cache v2
free space cache v2 cleared
$ sudo mount -o rw,subvolid=5,clear_cache,space_cache=v2,skip_balance,enospc_debug /dev/mapper/luks-fcd41acc-db81-4f2d-9e84-159b7c178fdf /mnt/sys
$ sudo touch /mnt/sys/test
touch: cannot touch '/mnt/sys/test': No space left on device
$ dmesg | grep -iE 'btrfs|enospc|space|abort|readonly|read-only|error|corrupt' | tail -200
[ 1319.679825] BTRFS info (device dm-0): chunk_block_rsv: size 0 reserved 0
[ 1319.679826] BTRFS info (device dm-0): delayed_block_rsv: size 0 reserved 0
[ 1319.679827] BTRFS info (device dm-0): delayed_refs_rsv: size 786432 reserved 0
[ 1319.679853] BTRFS info (device dm-0): space_info 4 has -245760 free, is full
[ 1319.679855] BTRFS info (device dm-0): space_info total=5368709120, used=4831920128, pinned=131072, reserved=16384, may_use=536821760, readonly=65536
[ 1319.679856] BTRFS info (device dm-0): global_block_rsv: size 536870912 reserved 536821760
[ 1319.679857] BTRFS info (device dm-0): trans_block_rsv: size 0 reserved 0
[ 1319.679858] BTRFS info (device dm-0): chunk_block_rsv: size 0 reserved 0
[ 1319.679859] BTRFS info (device dm-0): delayed_block_rsv: size 0 reserved 0
[ 1319.679861] BTRFS info (device dm-0): delayed_refs_rsv: size 1048576 reserved 0
[ 1319.679883] BTRFS info (device dm-0): space_info 4 has -245760 free, is full
[ 1319.679885] BTRFS info (device dm-0): space_info total=5368709120, used=4831903744, pinned=147456, reserved=32768, may_use=536805376, readonly=65536
[ 1319.679886] BTRFS info (device dm-0): global_block_rsv: size 536870912 reserved 536805376
[ 1319.679887] BTRFS info (device dm-0): trans_block_rsv: size 0 reserved 0
[ 1319.679888] BTRFS info (device dm-0): chunk_block_rsv: size 0 reserved 0
[ 1319.679889] BTRFS info (device dm-0): delayed_block_rsv: size 0 reserved 0
[ 1319.679891] BTRFS info (device dm-0): delayed_refs_rsv: size 1310720 reserved 0
[ 1319.679919] BTRFS info (device dm-0): space_info 4 has -245760 free, is full
[ 1319.679921] BTRFS info (device dm-0): space_info total=5368709120, used=4831887360, pinned=163840, reserved=49152, may_use=536788992, readonly=65536
[ 1319.679922] BTRFS info (device dm-0): global_block_rsv: size 536870912 reserved 536788992
[ 1319.679923] BTRFS info (device dm-0): trans_block_rsv: size 0 reserved 0
[ 1319.679924] BTRFS info (device dm-0): chunk_block_rsv: size 0 reserved 0
[ 1319.679926] BTRFS info (device dm-0): delayed_block_rsv: size 0 reserved 0
[ 1319.679927] BTRFS info (device dm-0): delayed_refs_rsv: size 1572864 reserved 0
[ 1319.679952] BTRFS info (device dm-0): space_info 4 has -245760 free, is full
[ 1319.679954] BTRFS info (device dm-0): space_info total=5368709120, used=4831870976, pinned=180224, reserved=65536, may_use=536772608, readonly=65536
[ 1319.679955] BTRFS info (device dm-0): global_block_rsv: size 536870912 reserved 536772608
[ 1319.679956] BTRFS info (device dm-0): trans_block_rsv: size 0 reserved 0
[ 1319.679957] BTRFS info (device dm-0): chunk_block_rsv: size 0 reserved 0
[ 1319.679959] BTRFS info (device dm-0): delayed_block_rsv: size 0 reserved 0
[ 1319.679960] BTRFS info (device dm-0): delayed_refs_rsv: size 1835008 reserved 0
[ 1319.680113] BTRFS info (device dm-0): space_info 4 has -245760 free, is full
[ 1319.680115] BTRFS info (device dm-0): space_info total=5368709120, used=4831903744, pinned=245760, reserved=16384, may_use=536723456, readonly=65536
[ 1319.680116] BTRFS info (device dm-0): global_block_rsv: size 536870912 reserved 536723456
[ 1319.680117] BTRFS info (device dm-0): trans_block_rsv: size 0 reserved 0
[ 1319.680118] BTRFS info (device dm-0): chunk_block_rsv: size 0 reserved 0
[ 1319.680119] BTRFS info (device dm-0): delayed_block_rsv: size 0 reserved 0
[ 1319.680120] BTRFS info (device dm-0): delayed_refs_rsv: size 786432 reserved 0
[ 1319.680160] BTRFS info (device dm-0): space_info 4 has -245760 free, is full
[ 1319.680162] BTRFS info (device dm-0): space_info total=5368709120, used=4831903744, pinned=278528, reserved=0, may_use=536707072, readonly=65536
[ 1319.680163] BTRFS info (device dm-0): global_block_rsv: size 536870912 reserved 536707072
[ 1319.680164] BTRFS info (device dm-0): trans_block_rsv: size 0 reserved 0
[ 1319.680165] BTRFS info (device dm-0): chunk_block_rsv: size 0 reserved 0
[ 1319.680166] BTRFS info (device dm-0): delayed_block_rsv: size 0 reserved 0
[ 1319.680167] BTRFS info (device dm-0): delayed_refs_rsv: size 524288 reserved 0
[ 1319.680187] BTRFS info (device dm-0): space_info 4 has -245760 free, is full
[ 1319.680188] BTRFS info (device dm-0): space_info total=5368709120, used=4831903744, pinned=278528, reserved=16384, may_use=536690688, readonly=65536
[ 1319.680189] BTRFS info (device dm-0): global_block_rsv: size 536870912 reserved 536690688
[ 1319.680190] BTRFS info (device dm-0): trans_block_rsv: size 0 reserved 0
[ 1319.680191] BTRFS info (device dm-0): chunk_block_rsv: size 0 reserved 0
[ 1319.680192] BTRFS info (device dm-0): delayed_block_rsv: size 0 reserved 0
[ 1319.680193] BTRFS info (device dm-0): delayed_refs_rsv: size 786432 reserved 0
[ 1319.680232] BTRFS info (device dm-0): space_info 4 has -245760 free, is full
[ 1319.680235] BTRFS info (device dm-0): space_info total=5368709120, used=4831936512, pinned=278528, reserved=0, may_use=536674304, readonly=65536
[ 1319.680236] BTRFS info (device dm-0): global_block_rsv: size 536870912 reserved 536674304
[ 1319.680237] BTRFS info (device dm-0): trans_block_rsv: size 0 reserved 0
[ 1319.680239] BTRFS info (device dm-0): chunk_block_rsv: size 0 reserved 0
[ 1319.680240] BTRFS info (device dm-0): delayed_block_rsv: size 0 reserved 0
[ 1319.680241] BTRFS info (device dm-0): delayed_refs_rsv: size 1048576 reserved 0
[ 1319.682077] BTRFS info (device dm-0): space_info 4 has -245760 free, is full
[ 1319.682079] BTRFS info (device dm-0): space_info total=5368709120, used=4831903744, pinned=163840, reserved=0, may_use=536821760, readonly=65536
[ 1319.682080] BTRFS info (device dm-0): global_block_rsv: size 536870912 reserved 536821760
[ 1319.682081] BTRFS info (device dm-0): trans_block_rsv: size 0 reserved 0
[ 1319.682082] BTRFS info (device dm-0): chunk_block_rsv: size 0 reserved 0
[ 1319.682083] BTRFS info (device dm-0): delayed_block_rsv: size 0 reserved 0
[ 1319.682083] BTRFS info (device dm-0): delayed_refs_rsv: size 524288 reserved 0
[ 3798.186224] BTRFS info (device dm-0): flagging fs with big metadata feature
[ 3798.186229] BTRFS info (device dm-0): force clearing of disk cache
[ 3798.186231] BTRFS info (device dm-0): enabling free space tree
[ 3798.186234] BTRFS info (device dm-0): using free space tree
[ 3798.186235] BTRFS info (device dm-0): has skinny extents
[ 3798.409903] BTRFS info (device dm-0): enabling ssd optimizations
[ 3798.411007] BTRFS info (device dm-0): creating free space tree
[ 3800.352026] BTRFS info (device dm-0): setting compat-ro feature flag for FREE_SPACE_TREE (0x1)
[ 3800.352028] BTRFS info (device dm-0): setting compat-ro feature flag for FREE_SPACE_TREE_VALID (0x2)
[ 3800.352872] BTRFS info (device dm-0): space_info 4 has -245760 free, is not full
[ 3800.352874] BTRFS info (device dm-0): space_info total=5368709120, used=4831592448, pinned=98304, reserved=360448, may_use=536838144, readonly=65536
[ 3800.352875] BTRFS info (device dm-0): global_block_rsv: size 536870912 reserved 536838144
[ 3800.352876] BTRFS info (device dm-0): trans_block_rsv: size 0 reserved 0
[ 3800.352878] BTRFS info (device dm-0): chunk_block_rsv: size 0 reserved 0
[ 3800.352879] BTRFS info (device dm-0): delayed_block_rsv: size 0 reserved 0
[ 3800.352880] BTRFS info (device dm-0): delayed_refs_rsv: size 6815744 reserved 0
[ 3800.352974] BTRFS info (device dm-0): space_info 4 has -245760 free, is not full
[ 3800.352976] BTRFS info (device dm-0): space_info total=5368709120, used=4831707136, pinned=131072, reserved=229376, may_use=536821760, readonly=65536
[ 3800.352978] BTRFS info (device dm-0): global_block_rsv: size 536870912 reserved 536821760
[ 3800.352979] BTRFS info (device dm-0): trans_block_rsv: size 0 reserved 0
[ 3800.352980] BTRFS info (device dm-0): chunk_block_rsv: size 0 reserved 0
[ 3800.352980] BTRFS info (device dm-0): delayed_block_rsv: size 0 reserved 0
[ 3800.352982] BTRFS info (device dm-0): delayed_refs_rsv: size 4456448 reserved 0
[ 3800.353018] BTRFS info (device dm-0): space_info 4 has -245760 free, is not full
[ 3800.353020] BTRFS info (device dm-0): space_info total=5368709120, used=4831772672, pinned=131072, reserved=180224, may_use=536805376, readonly=65536
[ 3800.353021] BTRFS info (device dm-0): global_block_rsv: size 536870912 reserved 536805376
[ 3800.353022] BTRFS info (device dm-0): trans_block_rsv: size 0 reserved 0
[ 3800.353023] BTRFS info (device dm-0): chunk_block_rsv: size 0 reserved 0
[ 3800.353024] BTRFS info (device dm-0): delayed_block_rsv: size 0 reserved 0
[ 3800.353025] BTRFS info (device dm-0): delayed_refs_rsv: size 3932160 reserved 0
[ 3800.353098] BTRFS info (device dm-0): space_info 4 has -245760 free, is not full
[ 3800.353100] BTRFS info (device dm-0): space_info total=5368709120, used=4831920128, pinned=131072, reserved=49152, may_use=536788992, readonly=65536
[ 3800.353101] BTRFS info (device dm-0): global_block_rsv: size 536870912 reserved 536788992
[ 3800.353102] BTRFS info (device dm-0): trans_block_rsv: size 0 reserved 0
[ 3800.353103] BTRFS info (device dm-0): chunk_block_rsv: size 0 reserved 0
[ 3800.353104] BTRFS info (device dm-0): delayed_block_rsv: size 0 reserved 0
[ 3800.353105] BTRFS info (device dm-0): delayed_refs_rsv: size 2097152 reserved 0
[ 3800.353142] BTRFS info (device dm-0): space_info 4 has -245760 free, is not full
[ 3800.353143] BTRFS info (device dm-0): space_info total=5368709120, used=4831985664, pinned=131072, reserved=0, may_use=536772608, readonly=65536
[ 3800.353145] BTRFS info (device dm-0): global_block_rsv: size 536870912 reserved 536772608
[ 3800.353146] BTRFS info (device dm-0): trans_block_rsv: size 0 reserved 0
[ 3800.353147] BTRFS info (device dm-0): chunk_block_rsv: size 0 reserved 0
[ 3800.353148] BTRFS info (device dm-0): delayed_block_rsv: size 0 reserved 0
[ 3800.353149] BTRFS info (device dm-0): delayed_refs_rsv: size 1572864 reserved 0
[ 3800.353173] BTRFS info (device dm-0): space_info 4 has -245760 free, is not full
[ 3800.353174] BTRFS info (device dm-0): space_info total=5368709120, used=4831952896, pinned=163840, reserved=16384, may_use=536756224, readonly=65536
[ 3800.353175] BTRFS info (device dm-0): global_block_rsv: size 536870912 reserved 536756224
[ 3800.353176] BTRFS info (device dm-0): trans_block_rsv: size 0 reserved 0
[ 3800.353177] BTRFS info (device dm-0): chunk_block_rsv: size 0 reserved 0
[ 3800.353178] BTRFS info (device dm-0): delayed_block_rsv: size 0 reserved 0
[ 3800.353179] BTRFS info (device dm-0): delayed_refs_rsv: size 1572864 reserved 0
[ 3800.353244] BTRFS info (device dm-0): space_info 4 has -245760 free, is not full
[ 3800.353246] BTRFS info (device dm-0): space_info total=5368709120, used=4831887360, pinned=245760, reserved=16384, may_use=536739840, readonly=65536
[ 3800.353247] BTRFS info (device dm-0): global_block_rsv: size 536870912 reserved 536739840
[ 3800.353248] BTRFS info (device dm-0): trans_block_rsv: size 0 reserved 0
[ 3800.353249] BTRFS info (device dm-0): chunk_block_rsv: size 0 reserved 0
[ 3800.353250] BTRFS info (device dm-0): delayed_block_rsv: size 0 reserved 0
[ 3800.353251] BTRFS info (device dm-0): delayed_refs_rsv: size 524288 reserved 0
[ 3800.353333] BTRFS info (device dm-0): space_info 4 has -245760 free, is not full
[ 3800.353335] BTRFS info (device dm-0): space_info total=5368709120, used=4831920128, pinned=262144, reserved=0, may_use=536707072, readonly=65536
[ 3800.353336] BTRFS info (device dm-0): global_block_rsv: size 536870912 reserved 536707072
[ 3800.353337] BTRFS info (device dm-0): trans_block_rsv: size 0 reserved 0
[ 3800.353338] BTRFS info (device dm-0): chunk_block_rsv: size 0 reserved 0
[ 3800.353339] BTRFS info (device dm-0): delayed_block_rsv: size 0 reserved 0
[ 3800.353340] BTRFS info (device dm-0): delayed_refs_rsv: size 524288 reserved 0
[ 3800.353440] BTRFS info (device dm-0): space_info 4 has -245760 free, is not full
[ 3800.353442] BTRFS info (device dm-0): space_info total=5368709120, used=4831887360, pinned=327680, reserved=16384, may_use=536657920, readonly=65536
[ 3800.353443] BTRFS info (device dm-0): global_block_rsv: size 536870912 reserved 536657920
[ 3800.353444] BTRFS info (device dm-0): trans_block_rsv: size 0 reserved 0
[ 3800.353445] BTRFS info (device dm-0): chunk_block_rsv: size 0 reserved 0
[ 3800.353446] BTRFS info (device dm-0): delayed_block_rsv: size 0 reserved 0
[ 3800.353447] BTRFS info (device dm-0): delayed_refs_rsv: size 524288 reserved 0
[ 3800.362391] BTRFS info (device dm-0): checking UUID tree
[ 3800.380718] BTRFS info (device dm-0): cannot satisfy tickets, dumping space info
[ 3800.380720] BTRFS info (device dm-0): space_info 4 has -131072 free, is full
[ 3800.380721] BTRFS info (device dm-0): space_info total=5368709120, used=4831903744, pinned=0, reserved=0, may_use=536870912, readonly=65536
[ 3800.380722] BTRFS info (device dm-0): global_block_rsv: size 536870912 reserved 536870912
[ 3800.380723] BTRFS info (device dm-0): trans_block_rsv: size 0 reserved 0
[ 3800.380723] BTRFS info (device dm-0): chunk_block_rsv: size 0 reserved 0
[ 3800.380724] BTRFS info (device dm-0): delayed_block_rsv: size 0 reserved 0
[ 3800.380725] BTRFS info (device dm-0): delayed_refs_rsv: size 0 reserved 0
[ 3800.380725] BTRFS info (device dm-0): failing ticket with 524288 bytes
[ 3800.380730] BTRFS info (device dm-0): space_info 4 has -131072 free, is full
[ 3800.380731] BTRFS info (device dm-0): space_info total=5368709120, used=4831903744, pinned=0, reserved=0, may_use=536870912, readonly=65536
[ 3800.380732] BTRFS info (device dm-0): global_block_rsv: size 536870912 reserved 536870912
[ 3800.380733] BTRFS info (device dm-0): trans_block_rsv: size 0 reserved 0
[ 3800.380733] BTRFS info (device dm-0): chunk_block_rsv: size 0 reserved 0
[ 3800.380734] BTRFS info (device dm-0): delayed_block_rsv: size 0 reserved 0
[ 3800.380735] BTRFS info (device dm-0): delayed_refs_rsv: size 0 reserved 0
[ 3800.380736] BTRFS warning (device dm-0): btrfs_uuid_scan_kthread failed -28
[ 3823.167395] BTRFS info (device dm-0): cannot satisfy tickets, dumping space info
[ 3823.167397] BTRFS info (device dm-0): space_info 4 has -131072 free, is full
[ 3823.167399] BTRFS info (device dm-0): space_info total=5368709120, used=4831903744, pinned=0, reserved=0, may_use=536870912, readonly=65536
[ 3823.167400] BTRFS info (device dm-0): global_block_rsv: size 536870912 reserved 536870912
[ 3823.167401] BTRFS info (device dm-0): trans_block_rsv: size 0 reserved 0
[ 3823.167402] BTRFS info (device dm-0): chunk_block_rsv: size 0 reserved 0
[ 3823.167402] BTRFS info (device dm-0): delayed_block_rsv: size 0 reserved 0
[ 3823.167403] BTRFS info (device dm-0): delayed_refs_rsv: size 0 reserved 0
[ 3823.167404] BTRFS info (device dm-0): failing ticket with 1310720 bytes
[ 3823.167440] BTRFS info (device dm-0): space_info 4 has -131072 free, is full
[ 3823.167442] BTRFS info (device dm-0): space_info total=5368709120, used=4831903744, pinned=0, reserved=0, may_use=536870912, readonly=65536
[ 3823.167443] BTRFS info (device dm-0): global_block_rsv: size 536870912 reserved 536870912
[ 3823.167444] BTRFS info (device dm-0): trans_block_rsv: size 0 reserved 0
[ 3823.167445] BTRFS info (device dm-0): chunk_block_rsv: size 0 reserved 0
[ 3823.167445] BTRFS info (device dm-0): delayed_block_rsv: size 0 reserved 0
[ 3823.167446] BTRFS info (device dm-0): delayed_refs_rsv: size 0 reserved 0
[ 3833.983254] BTRFS info (device dm-0): space_info 4 has -245760 free, is full
[ 3833.983260] BTRFS info (device dm-0): space_info total=5368709120, used=4831789056, pinned=114688, reserved=131072, may_use=536854528, readonly=65536
[ 3833.983263] BTRFS info (device dm-0): global_block_rsv: size 536870912 reserved 536854528
[ 3833.983265] BTRFS info (device dm-0): trans_block_rsv: size 0 reserved 0
[ 3833.983267] BTRFS info (device dm-0): chunk_block_rsv: size 0 reserved 0
[ 3833.983269] BTRFS info (device dm-0): delayed_block_rsv: size 0 reserved 0
[ 3833.983271] BTRFS info (device dm-0): delayed_refs_rsv: size 2621440 reserved 0
[ 3833.983361] BTRFS info (device dm-0): space_info 4 has -245760 free, is full
[ 3833.983364] BTRFS info (device dm-0): space_info total=5368709120, used=4831870976, pinned=114688, reserved=65536, may_use=536838144, readonly=65536
[ 3833.983367] BTRFS info (device dm-0): global_block_rsv: size 536870912 reserved 536838144
[ 3833.983369] BTRFS info (device dm-0): trans_block_rsv: size 0 reserved 0
[ 3833.983370] BTRFS info (device dm-0): chunk_block_rsv: size 0 reserved 0
[ 3833.983372] BTRFS info (device dm-0): delayed_block_rsv: size 0 reserved 0
[ 3833.983374] BTRFS info (device dm-0): delayed_refs_rsv: size 1835008 reserved 0
[ 3833.983491] BTRFS info (device dm-0): space_info 4 has -245760 free, is full
[ 3833.983494] BTRFS info (device dm-0): space_info total=5368709120, used=4831936512, pinned=131072, reserved=0, may_use=536821760, readonly=65536
[ 3833.983496] BTRFS info (device dm-0): global_block_rsv: size 536870912 reserved 536821760
[ 3833.983498] BTRFS info (device dm-0): trans_block_rsv: size 0 reserved 0
[ 3833.983500] BTRFS info (device dm-0): chunk_block_rsv: size 0 reserved 0
[ 3833.983502] BTRFS info (device dm-0): delayed_block_rsv: size 0 reserved 0
[ 3833.983504] BTRFS info (device dm-0): delayed_refs_rsv: size 786432 reserved 0
[ 3833.983675] BTRFS info (device dm-0): space_info 4 has -245760 free, is full
[ 3833.983678] BTRFS info (device dm-0): space_info total=5368709120, used=4831903744, pinned=196608, reserved=16384, may_use=536772608, readonly=65536
[ 3833.983680] BTRFS info (device dm-0): global_block_rsv: size 536870912 reserved 536772608
[ 3833.983682] BTRFS info (device dm-0): trans_block_rsv: size 0 reserved 0
[ 3833.983683] BTRFS info (device dm-0): chunk_block_rsv: size 0 reserved 0
[ 3833.983685] BTRFS info (device dm-0): delayed_block_rsv: size 0 reserved 0
[ 3833.983687] BTRFS info (device dm-0): delayed_refs_rsv: size 786432 reserved 0
$ sudo btrfs filesystem usage -T /mnt/sys
Overall:
    Device size:		 944.77GiB
    Device allocated:		 944.77GiB
    Device unallocated:		   1.00MiB
    Device missing:		     0.00B
    Used:			 844.33GiB
    Free (estimated):		  99.42GiB	(min: 99.42GiB)
    Data ratio:			      1.00
    Metadata ratio:		      2.00
    Global reserve:		 512.00MiB	(used: 0.00B)

                                                         Data      Metadata System               
Id Path                                                  single    DUP      DUP       Unallocated
-- ----------------------------------------------------- --------- -------- --------- -----------
 1 /dev/mapper/luks-fcd41acc-db81-4f2d-9e84-159b7c178fdf 934.75GiB 10.00GiB  16.00MiB     1.00MiB
-- ----------------------------------------------------- --------- -------- --------- -----------
   Total                                                 934.75GiB  5.00GiB   8.00MiB     1.00MiB
   Used                                                  835.33GiB  4.50GiB 128.00KiB            
$ sudo btrfs device usage /mnt/sys
/dev/mapper/luks-fcd41acc-db81-4f2d-9e84-159b7c178fdf, ID: 1
   Device size:           944.77GiB
   Device slack:              0.00B
   Data,single:           934.75GiB
   Metadata,DUP:           10.00GiB
   System,DUP:             16.00MiB
   Unallocated:             1.00MiB


$ sudo fdisk -l /dev/nvme0n1
Disk /dev/nvme0n1: 953.89 GiB, 1024209543168 bytes, 2000409264 sectors
Disk model: SSDPR-PX500-01T-80                      
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 31E18138-0E1C-441B-BB77-F0AE462EA509

Device              Start        End    Sectors   Size Type
/dev/nvme0n1p1       4096     618495     614400   300M EFI System
/dev/nvme0n1p2     618496 1981941759 1981323264 944.8G Linux filesystem
/dev/nvme0n1p3 1981942797 2000397734   18454938   8.8G Linux filesystem


$ sudo btrfs filesystem show /mnt/sys
Label: none  uuid: 9fa777d4-6ca8-4457-a1eb-eba91ac8eccd
	Total devices 1 FS bytes used 839.83GiB
	devid    1 size 944.77GiB used 944.77GiB path /dev/mapper/luks-fcd41acc-db81-4f2d-9e84-159b7c178fdf

$ sudo btrfs subvolume list -t /mnt/sys
ID	gen	top level	path	
--	---	---------	----	
256	1042851	5		@
257	1042483	5		@home
258	1040708	5		@cache
259	1042695	5		@log
3166	1043909	5		timeshift-btrfs/snapshots/2026-04-13_20-00-01/@
3167	767844	5		timeshift-btrfs/snapshots/2026-04-13_20-00-01/@home
3204	1042562	5		timeshift-btrfs/snapshots/2026-04-20_20-00-00/@
3205	779473	5		timeshift-btrfs/snapshots/2026-04-20_20-00-00/@home
3222	1042568	5		timeshift-btrfs/snapshots/2026-04-23_13-17-33/@
3223	784186	5		timeshift-btrfs/snapshots/2026-04-23_13-17-33/@home
3236	1042574	5		timeshift-btrfs/snapshots/2026-04-26_13-49-20/@
3237	787611	5		timeshift-btrfs/snapshots/2026-04-26_13-49-20/@home
3246	1042577	5		timeshift-btrfs/snapshots/2026-04-27_20-00-00/@
3247	790094	5		timeshift-btrfs/snapshots/2026-04-27_20-00-00/@home
3264	1042583	5		timeshift-btrfs/snapshots/2026-04-30_11-28-36/@
3265	794900	5		timeshift-btrfs/snapshots/2026-04-30_11-28-36/@home
3276	1042589	5		timeshift-btrfs/snapshots/2026-05-02_21-24-25/@
3277	797249	5		timeshift-btrfs/snapshots/2026-05-02_21-24-25/@home
3286	1050163	5		timeshift-btrfs/snapshots/2026-05-03_18-43-32/@
3287	798769	5		timeshift-btrfs/snapshots/2026-05-03_18-43-32/@home
3292	1042595	5		timeshift-btrfs/snapshots/2026-05-03_20-19-33/@
3293	799021	5		timeshift-btrfs/snapshots/2026-05-03_20-19-33/@home
3294	809118	5		timeshift-btrfs/snapshots/2026-05-03_23-00-00/@
3295	809118	5		timeshift-btrfs/snapshots/2026-05-03_23-00-00/@home



$ sudo btrfs balance start -dusage=15 /mnt/sys
Done, had to relocate 0 out of 943 chunks
$ sudo btrfs balance start -dusage=20 /mnt/sys
ERROR: error during balancing '/mnt/sys': No space left on device
There may be more info in syslog - try dmesg | tail
$ sudo btrfs balance start -musage=0 /mnt/sys
Done, had to relocate 0 out of 943 chunks
$ sudo btrfs balance start -musage=5 /mnt/sys
ERROR: error during balancing '/mnt/sys': No space left on device
There may be more info in syslog - try dmesg | tail

dmesg output, after write attempt (touch):

[ 1811.310469] BTRFS info (device dm-0): cannot satisfy tickets, dumping space info
[ 1811.310471] BTRFS info (device dm-0): space_info 4 has 262144 free, is full
[ 1811.310473] BTRFS info (device dm-0): space_info total=5368709120, used=4831510528, pinned=0, reserved=0, may_use=536870912, readonly=65536
[ 1811.310474] BTRFS info (device dm-0): global_block_rsv: size 536870912 reserved 536870912
[ 1811.310475] BTRFS info (device dm-0): trans_block_rsv: size 0 reserved 0
[ 1811.310476] BTRFS info (device dm-0): chunk_block_rsv: size 0 reserved 0
[ 1811.310476] BTRFS info (device dm-0): delayed_block_rsv: size 0 reserved 0
[ 1811.310477] BTRFS info (device dm-0): delayed_refs_rsv: size 0 reserved 0
[ 1811.310478] BTRFS info (device dm-0): failing ticket with 1310720 bytes
[ 1811.310519] BTRFS info (device dm-0): space_info 4 has 262144 free, is full
[ 1811.310521] BTRFS info (device dm-0): space_info total=5368709120, used=4831510528, pinned=0, reserved=0, may_use=536870912, readonly=65536
[ 1811.310522] BTRFS info (device dm-0): global_block_rsv: size 536870912 reserved 536870912
[ 1811.310523] BTRFS info (device dm-0): trans_block_rsv: size 0 reserved 0
[ 1811.310524] BTRFS info (device dm-0): chunk_block_rsv: size 0 reserved 0
[ 1811.310524] BTRFS info (device dm-0): delayed_block_rsv: size 0 reserved 0
[ 1811.310525] BTRFS info (device dm-0): delayed_refs_rsv: size 0 reserved 0

As an safe(probably stupid) attempt to fix it, I’ve tried to add an USB disk to the device, so that it can grow and re-ballance.

This has worked, in so much, that I can write to the btrfs volume again, but, I’ve tried to de-couple the devices some 3 hours ago, and it’s still running. (the dconvert line). Not sure if it will ever finish, but I’m pretty sure that if I’ll interrupt it, the entire volume is hosed.

$ sudo btrfs device add -f /dev/sdc /mnt/sys/
$ sudo btrfs balance start -musage=5 /mnt/sys
ERROR: error during balancing '/mnt/sys': No space left on device
There may be more info in syslog - try dmesg | tail
$ sudo btrfs balance start -musage=0 /mnt/sys
Done, had to relocate 0 out of 943 chunks
$ sudo btrfs balance start -dusage=15 /mnt/sys
Done, had to relocate 0 out of 943 chunks
$ sudo btrfs balance start -dusage=20 /mnt/sys
Done, had to relocate 4 out of 943 chunks
$ sudo touch /mnt/sys/test
$ sudo rm /mnt/sys/test
$ sudo btrfs balance start -musage=5 /mnt/sys
Done, had to relocate 1 out of 939 chunks
$ sudo btrfs device remove /dev/sdc /mnt/sys/
ERROR: error removing device '/dev/sdc': unable to go below two devices on raid1
$ sudo btrfs filesystem usage -T /mnt/sys/
Overall:
    Device size:		1002.07GiB
    Device allocated:		 940.81GiB
    Device unallocated:		  61.25GiB
    Device missing:		     0.00B
    Used:			 844.33GiB
    Free (estimated):		 156.68GiB	(min: 126.05GiB)
    Data ratio:			      1.00
    Metadata ratio:		      2.00
    Global reserve:		 512.00MiB	(used: 0.00B)

                                                         Data      Metadata System               
Id Path                                                  single    DUP      RAID1     Unallocated
-- ----------------------------------------------------- --------- -------- --------- -----------
 1 /dev/mapper/luks-fcd41acc-db81-4f2d-9e84-159b7c178fdf 930.75GiB 10.00GiB  32.00MiB     3.99GiB
 2 /dev/sdc                                                      -        -  32.00MiB    57.27GiB
-- ----------------------------------------------------- --------- -------- --------- -----------
   Total                                                 930.75GiB  5.00GiB  32.00MiB    61.25GiB
   Used                                                  835.33GiB  4.50GiB 128.00KiB            
$ sudo btrfs balance start -dconvert=single -mconvert=dup /mnt/sys/

My 2nd plan was to try delete one of the snapshot subvolumes (not yet sure how), but now it seems the volume might not even live that long.

If you have advice for me, I’d be glad. I might try to dd from the volume, but probably it’d be much better to just do a fresh install and use ext4.

I have no idea why btrfs stopped balancing, and that is my main reason for posting. Maybe your system is now hurling towards the same conclusion, so please, check your btrfs subvolumes, in my case, it was the metadata part, but the whole balance process seems to be a problem.

2

I would stop this right now!

Nothing is lost. But you will have to undo all this, and it will take just as long.

Plus you don’t want to add a flash stick as a single point of failure for your root file system.

Filling up a btrfs filesystem generates scary errors. Just don’t run scary commands after!

And this! Yes, btrfs defaults are single data, with dup metadata.

But that command is rewriting all data and all metadata out again (using dconvert).

The great thing about btrfs, is most likely nothing is lost, even having these commands with very unintended consequences.

This needs to be the first plan, this is most likely how we get our space back.

What snapshots, Timeshift?

We can delete them from the command line easily too. I can provide more info once you stop that balance.

If you want guidance on how to delete them. We will need the output of:

sudo btrfs sub list /mnt/sys

We do need to free up space first.

I am kind of lost where we’re at now though. You filled up your file system, and made it a little worse. But it sounds perfectly fixable.

If you want help, we kind of need to know what state it is in now.

sudo btrfs balance status /mnt/sys
# After all balances have stopped!

findmnt /mnt/sys
sudo btrfs filesystem usage /mnt/sys
sudo btrfs device stats /mnt/sys
sudo btrfs filesystem df /mnt/sys
4 Likes
3

Thanks for the answer, ehm, darn.

I think I can’t stop at the moment, as per the output of:

$ sudo btrfs filesystem usage -T /mnt/sys/
Overall:
    Device size:		1002.07GiB
    Device allocated:		 940.81GiB
    Device unallocated:		  61.25GiB
    Device missing:		     0.00B
    Used:			 844.33GiB
    Free (estimated):		 156.68GiB	(min: 126.05GiB)
    Data ratio:			      1.00
    Metadata ratio:		      2.00
    Global reserve:		 512.00MiB	(used: 0.00B)

                                                         Data      Metadata System               
Id Path                                                  single    DUP      RAID1     Unallocated
-- ----------------------------------------------------- --------- -------- --------- -----------
 1 /dev/mapper/luks-fcd41acc-db81-4f2d-9e84-159b7c178fdf 930.75GiB 10.00GiB  32.00MiB     3.99GiB
 2 /dev/sdc                                                      -        -  32.00MiB    57.27GiB
-- ----------------------------------------------------- --------- -------- --------- -----------
   Total                                                 930.75GiB  5.00GiB  32.00MiB    61.25GiB
   Used                                                  835.33GiB  4.50GiB 128.00KiB            

sudo btrfs balance start -dconvert=single -mconvert=dup /mnt/sys/

Has been running for over 3 hours now, I’m not sure if it will ever finish and if I’ll be able to run

sudo btrfs device remove /dev/sdc /mnt/sys/

after, but I think I can’t interrupt it without causing catastrophic damage to the volume?

As for the snapshots, would removing them from TTY calling the CLI version of timeshift work? I disregarded that, because I couldn’t preform any write operations, but I did not try it, I don’t know why.

Than I was concerned with removing the timeshift volumes with “btrfs subvolume delete”, because I thought snapshots were incemental, so I could bork all of the snapshots, is that incorrect? Which led me to the erroneous thought that the USB disk RAID was the safest option (in hindsight, it sound really stupid).

I have posted the output of

$ sudo btrfs subvolume list -t /mnt/sys

in the post above. I’ve run it again twice in short succession, with “btrfs balance start -dconvert=single -mconvert=dup” still running and without the column formatting flag.

The “gen” number is increasing, which, I guess, means that the operation is still running and will perhaps finish eventually?

$ sudo btrfs sub list /mnt/sys/
ID 256 gen 1063622 top level 5 path @
ID 257 gen 1063566 top level 5 path @home
ID 258 gen 1063426 top level 5 path @cache
ID 259 gen 1063538 top level 5 path @log
ID 3166 gen 1063622 top level 5 path timeshift-btrfs/snapshots/2026-04-13_20-00-01/@
ID 3167 gen 1063566 top level 5 path timeshift-btrfs/snapshots/2026-04-13_20-00-01/@home
ID 3204 gen 1063622 top level 5 path timeshift-btrfs/snapshots/2026-04-20_20-00-00/@
ID 3205 gen 1063566 top level 5 path timeshift-btrfs/snapshots/2026-04-20_20-00-00/@home
ID 3222 gen 1063622 top level 5 path timeshift-btrfs/snapshots/2026-04-23_13-17-33/@
ID 3223 gen 1063566 top level 5 path timeshift-btrfs/snapshots/2026-04-23_13-17-33/@home
ID 3236 gen 1063622 top level 5 path timeshift-btrfs/snapshots/2026-04-26_13-49-20/@
ID 3237 gen 1063566 top level 5 path timeshift-btrfs/snapshots/2026-04-26_13-49-20/@home
ID 3246 gen 1063622 top level 5 path timeshift-btrfs/snapshots/2026-04-27_20-00-00/@
ID 3247 gen 1063566 top level 5 path timeshift-btrfs/snapshots/2026-04-27_20-00-00/@home
ID 3264 gen 1063622 top level 5 path timeshift-btrfs/snapshots/2026-04-30_11-28-36/@
ID 3265 gen 1063566 top level 5 path timeshift-btrfs/snapshots/2026-04-30_11-28-36/@home
ID 3276 gen 1063622 top level 5 path timeshift-btrfs/snapshots/2026-05-02_21-24-25/@
ID 3277 gen 1063566 top level 5 path timeshift-btrfs/snapshots/2026-05-02_21-24-25/@home
ID 3286 gen 1063622 top level 5 path timeshift-btrfs/snapshots/2026-05-03_18-43-32/@
ID 3287 gen 1063566 top level 5 path timeshift-btrfs/snapshots/2026-05-03_18-43-32/@home
ID 3292 gen 1063622 top level 5 path timeshift-btrfs/snapshots/2026-05-03_20-19-33/@
ID 3293 gen 1063566 top level 5 path timeshift-btrfs/snapshots/2026-05-03_20-19-33/@home
ID 3294 gen 1063622 top level 5 path timeshift-btrfs/snapshots/2026-05-03_23-00-00/@
ID 3295 gen 1063566 top level 5 path timeshift-btrfs/snapshots/2026-05-03_23-00-00/@home

$ sudo btrfs sub list /mnt/sys/
ID 256 gen 1063677 top level 5 path @
ID 257 gen 1063668 top level 5 path @home
ID 258 gen 1063426 top level 5 path @cache
ID 259 gen 1063538 top level 5 path @log
ID 3166 gen 1063677 top level 5 path timeshift-btrfs/snapshots/2026-04-13_20-00-01/@
ID 3167 gen 1063668 top level 5 path timeshift-btrfs/snapshots/2026-04-13_20-00-01/@home
ID 3204 gen 1063677 top level 5 path timeshift-btrfs/snapshots/2026-04-20_20-00-00/@
ID 3205 gen 1063668 top level 5 path timeshift-btrfs/snapshots/2026-04-20_20-00-00/@home
ID 3222 gen 1063677 top level 5 path timeshift-btrfs/snapshots/2026-04-23_13-17-33/@
ID 3223 gen 1063668 top level 5 path timeshift-btrfs/snapshots/2026-04-23_13-17-33/@home
ID 3236 gen 1063677 top level 5 path timeshift-btrfs/snapshots/2026-04-26_13-49-20/@
ID 3237 gen 1063668 top level 5 path timeshift-btrfs/snapshots/2026-04-26_13-49-20/@home
ID 3246 gen 1063677 top level 5 path timeshift-btrfs/snapshots/2026-04-27_20-00-00/@
ID 3247 gen 1063668 top level 5 path timeshift-btrfs/snapshots/2026-04-27_20-00-00/@home
ID 3264 gen 1063677 top level 5 path timeshift-btrfs/snapshots/2026-04-30_11-28-36/@
ID 3265 gen 1063668 top level 5 path timeshift-btrfs/snapshots/2026-04-30_11-28-36/@home
ID 3276 gen 1063677 top level 5 path timeshift-btrfs/snapshots/2026-05-02_21-24-25/@
ID 3277 gen 1063668 top level 5 path timeshift-btrfs/snapshots/2026-05-02_21-24-25/@home
ID 3286 gen 1063677 top level 5 path timeshift-btrfs/snapshots/2026-05-03_18-43-32/@
ID 3287 gen 1063668 top level 5 path timeshift-btrfs/snapshots/2026-05-03_18-43-32/@home
ID 3292 gen 1063677 top level 5 path timeshift-btrfs/snapshots/2026-05-03_20-19-33/@
ID 3293 gen 1063668 top level 5 path timeshift-btrfs/snapshots/2026-05-03_20-19-33/@home
ID 3294 gen 1063677 top level 5 path timeshift-btrfs/snapshots/2026-05-03_23-00-00/@
ID 3295 gen 1063668 top level 5 path timeshift-btrfs/snapshots/2026-05-03_23-00-00/@home
5

Nope. Cancel, pause, or reboot at any time.

sudo btrfs balance cancel /mnt/sys/

I have to AFK, but this is perfectly fixable.

3 Likes
6

One thing at a time. But first things first:

You need to get out of panic mode!

Otherwise, you’ll break your system.

Please remember, btrfs is not a newer version of ext4. There are fundamental differences. Please take the time NOW to read the btrfs page in the Manjaro wiki. Not the whole page (maybe later), but definitely the section about out-of-space problems.

You will also find a step-by-step guide there on how to resolve the issue.

However, you should only attempt the USB stick method if it is absolutely necessary (i.e., if the btrfs log reports an “out of space” error, and btrfs usage shows that there are no free blocks left).

Running a balance operation is only permissible once you have alleviated the space issue by deleting files—and, above all (!!!), by removing the 2 or 3 oldest snapshots.

Even then, you should first wait for btrfs to release the space. You can verify this over the course of a few minutes by repeatedly running the btrfs filesystem usage command.

It is absolutely crucial that you remove the oldest ones; otherwise, it will be of little to no use at all!!!

By the way, you can also delete snapshots without any issues while a balance is running. (btrfs is magical. :footprints: )

You just have to keep in mind that a reboot won’t actually stop the balance. After the reboot, the balance simply picks up right where it left off. And if you started a balance without any filters—and you have a hard drive—it could take days! :wink:

:warning:
And one final warning: Once you have started the process involving the USB stick, please do not get the idea to unplug it. Under no circumstances. Not even during a
reboot. It absolutely must remain connected until you have seen this through.
:footprints:

4 Likes
7

Thank you to both @andreas85 , @Molski, I have cancelled the operation, it was clearly going the wrong way, transferring files to the USB.

$ sudo btrfs balance start -dconvert=single -mconvert=dup /mnt/sys/

balance canceled by user
$ 
$ sudo btrfs filesystem usage -T /mnt/sys/
Overall:
    Device size:		1002.07GiB
    Device allocated:		 847.07GiB
    Device unallocated:		 155.00GiB
    Device missing:		     0.00B
    Used:			 844.25GiB
    Free (estimated):		 156.68GiB	(min: 79.18GiB)
    Data ratio:			      1.00
    Metadata ratio:		      2.00
    Global reserve:		 512.00MiB	(used: 0.00B)

                                                         Data      Metadata System               
Id Path                                                  single    DUP      DUP       Unallocated
-- ----------------------------------------------------- --------- -------- --------- -----------
 1 /dev/mapper/luks-fcd41acc-db81-4f2d-9e84-159b7c178fdf 815.01GiB 10.00GiB         -   119.76GiB
 2 /dev/sdc                                               22.00GiB        -  64.00MiB    35.24GiB
-- ----------------------------------------------------- --------- -------- --------- -----------
   Total                                                 837.01GiB  5.00GiB  32.00MiB   155.00GiB
   Used                                                  835.33GiB  4.46GiB 128.00KiB            
$ sudo btrfs sub list /mnt/sys/
ID 256 gen 1064331 top level 5 path @
ID 257 gen 1064331 top level 5 path @home
ID 258 gen 1064331 top level 5 path @cache
ID 259 gen 1064311 top level 5 path @log
ID 3166 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-04-13_20-00-01/@
ID 3167 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-04-13_20-00-01/@home
ID 3204 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-04-20_20-00-00/@
ID 3205 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-04-20_20-00-00/@home
ID 3222 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-04-23_13-17-33/@
ID 3223 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-04-23_13-17-33/@home
ID 3236 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-04-26_13-49-20/@
ID 3237 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-04-26_13-49-20/@home
ID 3246 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-04-27_20-00-00/@
ID 3247 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-04-27_20-00-00/@home
ID 3264 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-04-30_11-28-36/@
ID 3265 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-04-30_11-28-36/@home
ID 3276 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-05-02_21-24-25/@
ID 3277 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-05-02_21-24-25/@home
ID 3286 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-05-03_18-43-32/@
ID 3287 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-05-03_18-43-32/@home
ID 3292 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-05-03_20-19-33/@
ID 3293 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-05-03_20-19-33/@home
ID 3294 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-05-03_23-00-00/@
ID 3295 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-05-03_23-00-00/@home

Can I just do the subvolume delete like this, with this exact syntax?

sudo btrfs subvolume delete /mnt/sys/timeshift-btrfs/snapshots/2026-04-13_20-00-01/@
sudo btrfs subvolume delete /mnt/sys/timeshift-btrfs/snapshots/2026-04-13_20-00-01/@home
sudo btrfs subvolume delete /mnt/sys/timeshift-btrfs/snapshots/2026-04-20_20-00-00/@
sudo btrfs subvolume delete /mnt/sys/timeshift-btrfs/snapshots/2026-04-20_20-00-00/@home
sudo btrfs subvolume delete /mnt/sys/timeshift-btrfs/snapshots/2026-04-23_13-17-33/@
sudo btrfs subvolume delete /mnt/sys/timeshift-btrfs/snapshots/2026-04-23_13-17-33/@home

Also, once there, how do I get rid of the USB, or to use the correct term, how do I make my volume consist of one device again?

I’ve tried

sudo btrfs device remove /dev/sdc /mnt/sys/

It’s the same as wiki suggests, but instead of:

This will not delete any data, but remove the device from the volume. Bevorehand all data will be copied to the remaining devices of the volume.

I got:

ERROR: error removing device ‘/dev/sdc’: unable to go below two devices on raid1

In the meantime, I will keep on reading and update this post. Not doing any changes to the disk atm.

8

If you have more than twice as much unallocated space as the size of the USB stick +10GB , you can have it removed from the file system.

Run one of them (the oldest one), and btrfs will tell you how good your performance was :wink:

Afterwards, use usage to check how the storage is being freed up.

Did you tinker with the balance operations regarding the RAID levels for your files or metadata?

If so, you need to revert those changes. The settings must be restored to their defaults: Data: single, Metadata: dup.

If you need to run a balance operation to achieve this, please make sure to use a filter! The filter should be configured to rewrite only those chunks that currently have the incorrect profile.

According to this output, systemdata is labeled as RAID1. However, it absolutely needs to be DUP. I don’t know how you managed to do that, but that is the reason why you cannot remove the USB stick.

But don’t panic. You can leave the USB stick connected for days if necessary. As previously mentioned, even a system reboot won’t cause any issues. Furthermore, you can continue to use the system—quite normally—as soon as sufficient free space becomes available.

Once everything is back in order, Btrfs will allow you to safely remove the USB stick again.

4 Likes
9

Oh, this seems promising, I started mashing in the commands where your response ended with

Run one of them (the oldest one), and btrfs will tell you how good your performance was

Output is below.

As for the RAID levels, I did not mess with them, the entire output saga of adding the USB and then btrfs mentioning RAID during removal attempt is this:

$ sudo btrfs device add -f /dev/sdc /mnt/sys/
$ sudo btrfs balance start -musage=5 /mnt/sys
ERROR: error during balancing '/mnt/sys': No space left on device
There may be more info in syslog - try dmesg | tail
$ sudo btrfs balance start -musage=0 /mnt/sys
Done, had to relocate 0 out of 943 chunks
$ sudo btrfs balance start -dusage=15 /mnt/sys
Done, had to relocate 0 out of 943 chunks
$ sudo btrfs balance start -dusage=20 /mnt/sys
Done, had to relocate 4 out of 943 chunks
$ sudo touch /mnt/sys/test
$ sudo rm /mnt/sys/test
$ sudo btrfs balance start -musage=5 /mnt/sys
Done, had to relocate 1 out of 939 chunks
$ sudo btrfs device remove /dev/sdc /mnt/sys/
ERROR: error removing device '/dev/sdc': unable to go below two devices on raid1

As for the attempt to delete 3 oldest snapshots, I did that, got from 156Gigs of free space to 219. USB is 64 GB, so both should have been sufficient.


$ sudo btrfs balance start -dconvert=single -mconvert=dup /mnt/sys/

balance canceled by user
$ 
$ sudo btrfs filesystem usage -T /mnt/sys/
Overall:
    Device size:		1002.07GiB
    Device allocated:		 847.07GiB
    Device unallocated:		 155.00GiB
    Device missing:		     0.00B
    Used:			 844.25GiB
    Free (estimated):		 156.68GiB	(min: 79.18GiB)
    Data ratio:			      1.00
    Metadata ratio:		      2.00
    Global reserve:		 512.00MiB	(used: 0.00B)

                                                         Data      Metadata System               
Id Path                                                  single    DUP      DUP       Unallocated
-- ----------------------------------------------------- --------- -------- --------- -----------
 1 /dev/mapper/luks-fcd41acc-db81-4f2d-9e84-159b7c178fdf 815.01GiB 10.00GiB         -   119.76GiB
 2 /dev/sdc                                               22.00GiB        -  64.00MiB    35.24GiB
-- ----------------------------------------------------- --------- -------- --------- -----------
   Total                                                 837.01GiB  5.00GiB  32.00MiB   155.00GiB
   Used                                                  835.33GiB  4.46GiB 128.00KiB            
$ sudo btrfs sub list /mnt/sys/
ID 256 gen 1064331 top level 5 path @
ID 257 gen 1064331 top level 5 path @home
ID 258 gen 1064331 top level 5 path @cache
ID 259 gen 1064311 top level 5 path @log
ID 3166 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-04-13_20-00-01/@
ID 3167 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-04-13_20-00-01/@home
ID 3204 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-04-20_20-00-00/@
ID 3205 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-04-20_20-00-00/@home
ID 3222 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-04-23_13-17-33/@
ID 3223 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-04-23_13-17-33/@home
ID 3236 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-04-26_13-49-20/@
ID 3237 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-04-26_13-49-20/@home
ID 3246 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-04-27_20-00-00/@
ID 3247 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-04-27_20-00-00/@home
ID 3264 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-04-30_11-28-36/@
ID 3265 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-04-30_11-28-36/@home
ID 3276 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-05-02_21-24-25/@
ID 3277 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-05-02_21-24-25/@home
ID 3286 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-05-03_18-43-32/@
ID 3287 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-05-03_18-43-32/@home
ID 3292 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-05-03_20-19-33/@
ID 3293 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-05-03_20-19-33/@home
ID 3294 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-05-03_23-00-00/@
ID 3295 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-05-03_23-00-00/@home


$ sudo btrfs filesystem usage /mnt/sys 
Overall:
    Device size:		1002.07GiB
    Device allocated:		 847.07GiB
    Device unallocated:		 155.00GiB
    Device missing:		     0.00B
    Used:			 844.25GiB
    Free (estimated):		 156.68GiB	(min: 79.18GiB)
    Data ratio:			      1.00
    Metadata ratio:		      2.00
    Global reserve:		 512.00MiB	(used: 0.00B)

Data,single: Size:837.01GiB, Used:835.33GiB (99.80%)
   /dev/mapper/luks-fcd41acc-db81-4f2d-9e84-159b7c178fdf	 815.01GiB
   /dev/sdc	  22.00GiB

Metadata,DUP: Size:5.00GiB, Used:4.46GiB (89.19%)
   /dev/mapper/luks-fcd41acc-db81-4f2d-9e84-159b7c178fdf	  10.00GiB

System,DUP: Size:32.00MiB, Used:128.00KiB (0.39%)
   /dev/sdc	  64.00MiB

Unallocated:
   /dev/mapper/luks-fcd41acc-db81-4f2d-9e84-159b7c178fdf	 119.76GiB
   /dev/sdc	  35.24GiB
$ sudo btrfs subvolume delete /mnt/sys/timeshift-btrfs/snapshots/2026-04-13_20-00-01/@
Delete subvolume (no-commit): '/mnt/sys/timeshift-btrfs/snapshots/2026-04-13_20-00-01/@'
$ sudo btrfs sub list /mnt/sys/
ID 256 gen 1064331 top level 5 path @
ID 257 gen 1064331 top level 5 path @home
ID 258 gen 1064331 top level 5 path @cache
ID 259 gen 1064311 top level 5 path @log
ID 3167 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-04-13_20-00-01/@home
ID 3204 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-04-20_20-00-00/@
ID 3205 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-04-20_20-00-00/@home
ID 3222 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-04-23_13-17-33/@
ID 3223 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-04-23_13-17-33/@home
ID 3236 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-04-26_13-49-20/@
ID 3237 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-04-26_13-49-20/@home
ID 3246 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-04-27_20-00-00/@
ID 3247 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-04-27_20-00-00/@home
ID 3264 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-04-30_11-28-36/@
ID 3265 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-04-30_11-28-36/@home
ID 3276 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-05-02_21-24-25/@
ID 3277 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-05-02_21-24-25/@home
ID 3286 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-05-03_18-43-32/@
ID 3287 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-05-03_18-43-32/@home
ID 3292 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-05-03_20-19-33/@
ID 3293 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-05-03_20-19-33/@home
ID 3294 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-05-03_23-00-00/@
ID 3295 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-05-03_23-00-00/@home
$ sudo btrfs filesystem usage /mnt/sys 
Overall:
    Device size:		1002.07GiB
    Device allocated:		 847.07GiB
    Device unallocated:		 155.00GiB
    Device missing:		     0.00B
    Used:			 844.25GiB
    Free (estimated):		 156.68GiB	(min: 79.18GiB)
    Data ratio:			      1.00
    Metadata ratio:		      2.00
    Global reserve:		 512.00MiB	(used: 48.00KiB)

Data,single: Size:837.01GiB, Used:835.33GiB (99.80%)
   /dev/mapper/luks-fcd41acc-db81-4f2d-9e84-159b7c178fdf	 815.01GiB
   /dev/sdc	  22.00GiB

Metadata,DUP: Size:5.00GiB, Used:4.46GiB (89.19%)
   /dev/mapper/luks-fcd41acc-db81-4f2d-9e84-159b7c178fdf	  10.00GiB

System,DUP: Size:32.00MiB, Used:128.00KiB (0.39%)
   /dev/sdc	  64.00MiB

Unallocated:
   /dev/mapper/luks-fcd41acc-db81-4f2d-9e84-159b7c178fdf	 119.76GiB
   /dev/sdc	  35.24GiB
$ sudo btrfs subvolume delete /mnt/sys/timeshift-btrfs/snapshots/2026-04-13_20-00-01/@home
Delete subvolume (no-commit): '/mnt/sys/timeshift-btrfs/snapshots/2026-04-13_20-00-01/@home'
$ sudo btrfs subvolume delete /mnt/sys/timeshift-btrfs/snapshots/2026-04-20_20-00-00/@
Delete subvolume (no-commit): '/mnt/sys/timeshift-btrfs/snapshots/2026-04-20_20-00-00/@'
$ sudo btrfs subvolume delete /mnt/sys/timeshift-btrfs/2026-04-20_20-00-00/@home
ERROR: Could not statfs: No such file or directory
$ sudo btrfs subvolume delete /mnt/sys/timeshift-btrfs/snapshots/2026-04-20_20-00-00/@home
Delete subvolume (no-commit): '/mnt/sys/timeshift-btrfs/snapshots/2026-04-20_20-00-00/@home'
$ sudo btrfs subvolume delete /mnt/sys/timeshift-btrfs/snapshots/2026-04-23_13-17-33/@
Delete subvolume (no-commit): '/mnt/sys/timeshift-btrfs/snapshots/2026-04-23_13-17-33/@'
$ sudo btrfs subvolume delete /mnt/sys/timeshift-btrfs/snapshots/2026-04-23_13-17-33/@home
Delete subvolume (no-commit): '/mnt/sys/timeshift-btrfs/snapshots/2026-04-23_13-17-33/@home'
$ sudo btrfs filesystem usage /mnt/sys 
Overall:
    Device size:		1002.07GiB
    Device allocated:		 847.07GiB
    Device unallocated:		 155.00GiB
    Device missing:		     0.00B
    Used:			 780.30GiB
    Free (estimated):		 219.31GiB	(min: 141.82GiB)
    Data ratio:			      1.00
    Metadata ratio:		      2.00
    Global reserve:		 512.00MiB	(used: 48.00KiB)

Data,single: Size:837.01GiB, Used:772.69GiB (92.32%)
   /dev/mapper/luks-fcd41acc-db81-4f2d-9e84-159b7c178fdf	 815.01GiB
   /dev/sdc	  22.00GiB

Metadata,DUP: Size:5.00GiB, Used:3.81GiB (76.14%)
   /dev/mapper/luks-fcd41acc-db81-4f2d-9e84-159b7c178fdf	  10.00GiB

System,DUP: Size:32.00MiB, Used:128.00KiB (0.39%)
   /dev/sdc	  64.00MiB

Unallocated:
   /dev/mapper/luks-fcd41acc-db81-4f2d-9e84-159b7c178fdf	 119.76GiB
   /dev/sdc	  35.24GiB
$ sudo btrfs sub list /mnt/sys/
ID 256 gen 1064331 top level 5 path @
ID 257 gen 1064331 top level 5 path @home
ID 258 gen 1064331 top level 5 path @cache
ID 259 gen 1064311 top level 5 path @log
ID 3236 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-04-26_13-49-20/@
ID 3237 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-04-26_13-49-20/@home
ID 3246 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-04-27_20-00-00/@
ID 3247 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-04-27_20-00-00/@home
ID 3264 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-04-30_11-28-36/@
ID 3265 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-04-30_11-28-36/@home
ID 3276 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-05-02_21-24-25/@
ID 3277 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-05-02_21-24-25/@home
ID 3286 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-05-03_18-43-32/@
ID 3287 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-05-03_18-43-32/@home
ID 3292 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-05-03_20-19-33/@
ID 3293 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-05-03_20-19-33/@home
ID 3294 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-05-03_23-00-00/@
ID 3295 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-05-03_23-00-00/@home
$ ^C
$ sudo btrfs device remove /dev/sdc /mnt/sys/

And as I was writing this poste, the “btrfs device remove” finished, USB is not mentioned in volume devices and system is still writeable:



$ sudo btrfs device remove /dev/sdc /mnt/sys/
$ sudo btrfs sub list /mnt/sys/
ID 256 gen 1064526 top level 5 path @
ID 257 gen 1064331 top level 5 path @home
ID 258 gen 1064508 top level 5 path @cache
ID 259 gen 1064311 top level 5 path @log
ID 3236 gen 1064535 top level 5 path timeshift-btrfs/snapshots/2026-04-26_13-49-20/@
ID 3237 gen 1064444 top level 5 path timeshift-btrfs/snapshots/2026-04-26_13-49-20/@home
ID 3246 gen 1064535 top level 5 path timeshift-btrfs/snapshots/2026-04-27_20-00-00/@
ID 3247 gen 1064444 top level 5 path timeshift-btrfs/snapshots/2026-04-27_20-00-00/@home
ID 3264 gen 1064535 top level 5 path timeshift-btrfs/snapshots/2026-04-30_11-28-36/@
ID 3265 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-04-30_11-28-36/@home
ID 3276 gen 1064535 top level 5 path timeshift-btrfs/snapshots/2026-05-02_21-24-25/@
ID 3277 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-05-02_21-24-25/@home
ID 3286 gen 1064535 top level 5 path timeshift-btrfs/snapshots/2026-05-03_18-43-32/@
ID 3287 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-05-03_18-43-32/@home
ID 3292 gen 1064535 top level 5 path timeshift-btrfs/snapshots/2026-05-03_20-19-33/@
ID 3293 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-05-03_20-19-33/@home
ID 3294 gen 1064526 top level 5 path timeshift-btrfs/snapshots/2026-05-03_23-00-00/@
ID 3295 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-05-03_23-00-00/@home
$ sudo btrfs filesystem usage /mnt/sys 
Overall:
    Device size:		 944.77GiB
    Device allocated:		 825.07GiB
    Device unallocated:		 119.70GiB
    Device missing:		     0.00B
    Used:			 766.72GiB
    Free (estimated):		 174.43GiB	(min: 114.58GiB)
    Data ratio:			      1.00
    Metadata ratio:		      2.00
    Global reserve:		 512.00MiB	(used: 0.00B)

Data,single: Size:815.01GiB, Used:760.28GiB (93.28%)
   /dev/mapper/luks-fcd41acc-db81-4f2d-9e84-159b7c178fdf	 815.01GiB

Metadata,DUP: Size:5.00GiB, Used:3.22GiB (64.46%)
   /dev/mapper/luks-fcd41acc-db81-4f2d-9e84-159b7c178fdf	  10.00GiB

System,DUP: Size:32.00MiB, Used:128.00KiB (0.39%)
   /dev/mapper/luks-fcd41acc-db81-4f2d-9e84-159b7c178fdf	  64.00MiB

Unallocated:
   /dev/mapper/luks-fcd41acc-db81-4f2d-9e84-159b7c178fdf	 119.70GiB
$ sudo btrfs filesystem usage -T /mnt/sys/
Overall:
    Device size:		 944.77GiB
    Device allocated:		 825.07GiB
    Device unallocated:		 119.70GiB
    Device missing:		     0.00B
    Used:			 766.72GiB
    Free (estimated):		 174.43GiB	(min: 114.58GiB)
    Data ratio:			      1.00
    Metadata ratio:		      2.00
    Global reserve:		 512.00MiB	(used: 0.00B)

                                                         Data      Metadata System               
Id Path                                                  single    DUP      DUP       Unallocated
-- ----------------------------------------------------- --------- -------- --------- -----------
 1 /dev/mapper/luks-fcd41acc-db81-4f2d-9e84-159b7c178fdf 815.01GiB 10.00GiB  64.00MiB   119.70GiB
-- ----------------------------------------------------- --------- -------- --------- -----------
   Total                                                 815.01GiB  5.00GiB  32.00MiB   119.70GiB
   Used                                                  760.28GiB  3.22GiB 128.00KiB            
$ sudo touch /mnt/sys/test
$ sudo rm /mnt/sys/test

Can I eject the USB now and boot into Manjaro?

10

I would leave it in place while you restart (just in case).

:notebook:
Oh, and by the way, it would be nice if you mentioned the actual reason (filesystem full / out of space) in the thread title. At the moment, the title is misleading.

1 Like
11

First make sure it’s not mounted.

12

I’ve rebooted with USB still in.

In grub, I can still see the old snapshots, I can also see them in timeshift, but in dolphin, the whole folder only have around 1KB. Should I delete them in timeshift?

The USB is, I believe, not mounted, it has no mountpoint:

lsblk -o NAME,PATH,FSTYPE,SIZE,MOUNTPOINTS                                                                                                                                  
NAME                                          PATH                                                  FSTYPE        SIZE MOUNTPOINTS
sda                                           /dev/sda                                                           57.3G 
nvme0n1                                       /dev/nvme0n1                                                      953.9G 
├─nvme0n1p1                                   /dev/nvme0n1p1                                        vfat          300M /boot/efi
├─nvme0n1p2                                   /dev/nvme0n1p2                                        crypto_LUKS 944.8G 
│ └─luks-fcd41acc-db81-4f2d-9e84-159b7c178fdf /dev/mapper/luks-fcd41acc-db81-4f2d-9e84-159b7c178fdf btrfs       944.8G /var/log
│                                                                                                                      /var/cache
│                                                                                                                      /home
│                                                                                                                      /
└─nvme0n1p3                                   /dev/nvme0n1p3                                        crypto_LUKS   8.8G 
  └─luks-e8d44145-619b-4c4a-b82b-75a7b2ff71e8 /dev/mapper/luks-e8d44145-619b-4c4a-b82b-75a7b2ff71e8 swap          8.8G [SWAP]

I have tried to go to different snapshots from grub, but it seems no rollback has happened. After the update, the pam.d for kde was moved and restored to original, but the sudoedit I did after update is still there. Makes sense, the system couldn’t be written to, but it’s a bit strange, because the TTY I got into was definitely from the older version (older kernel, etc.), but the wiki says that snapshots could be read, as in read-only, so maybe that’s what I was seeing?

I still have no idea what has happened and how to stop it from happening. Before the update, I’ve checked free space and had over 160GB free.

After update I didn’t really check until later from tty, but still, there was 99.42GiB free. At the moment, system is looking like this:

sudo btrfs filesystem usage -T /      
Overall:
    Device size:                 944.77GiB
    Device allocated:            825.07GiB
    Device unallocated:          119.70GiB
    Device missing:                  0.00B
    Device slack:                    0.00B
    Used:                        766.57GiB
    Free (estimated):            174.33GiB      (min: 114.48GiB)
    Free (statfs, df):           174.32GiB
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:              512.00MiB      (used: 0.00B)
    Multiple profiles:                  no

                                                         Data      Metadata System                               
Id Path                                                  single    DUP      DUP       Unallocated Total     Slack
-- ----------------------------------------------------- --------- -------- --------- ----------- --------- -----
 1 /dev/mapper/luks-fcd41acc-db81-4f2d-9e84-159b7c178fdf 815.01GiB 10.00GiB  64.00MiB   119.70GiB 944.77GiB     -
-- ----------------------------------------------------- --------- -------- --------- ----------- --------- -----
   Total                                                 815.01GiB  5.00GiB  32.00MiB   119.70GiB 944.77GiB 0.00B
   Used                                                  760.38GiB  3.10GiB 128.00KiB                            
sudo btrfs sub list /                                                                                                                                                                                                                                                                                                                                                       ID 256 gen 1064590 top level 5 path @
ID 257 gen 1064590 top level 5 path @home
ID 258 gen 1064558 top level 5 path @cache
ID 259 gen 1064590 top level 5 path @log
ID 3236 gen 1064557 top level 5 path timeshift-btrfs/snapshots/2026-04-26_13-49-20/@
ID 3237 gen 1064557 top level 5 path timeshift-btrfs/snapshots/2026-04-26_13-49-20/@home
ID 3246 gen 1064556 top level 5 path timeshift-btrfs/snapshots/2026-04-27_20-00-00/@
ID 3247 gen 1064444 top level 5 path timeshift-btrfs/snapshots/2026-04-27_20-00-00/@home
ID 3264 gen 1064556 top level 5 path timeshift-btrfs/snapshots/2026-04-30_11-28-36/@
ID 3265 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-04-30_11-28-36/@home
ID 3276 gen 1064556 top level 5 path timeshift-btrfs/snapshots/2026-05-02_21-24-25/@
ID 3277 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-05-02_21-24-25/@home
ID 3286 gen 1064556 top level 5 path timeshift-btrfs/snapshots/2026-05-03_18-43-32/@
ID 3287 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-05-03_18-43-32/@home
ID 3294 gen 1064556 top level 5 path timeshift-btrfs/snapshots/2026-05-03_23-00-00/@
ID 3295 gen 1064331 top level 5 path timeshift-btrfs/snapshots/2026-05-03_23-00-00/@home
ID 4351 gen 1064570 top level 5 path timeshift-btrfs/snapshots/2026-05-04_19-43-32/@
ID 4352 gen 1064571 top level 5 path timeshift-btrfs/snapshots/2026-05-04_19-43-32/@home

There was no warning, no notification, just the original plasma crash and then filesystem unwriteable.

True, it could have been a pretty quick fix, but I still have no idea how to avoid this, apart from leaving much larger reserve than the 15-20%.

And the thing that was full was metadata, not the stored data. How do I monitor that? Will they stay balanced in the future?

e: In the meantime, system did new daily and boot snapshot and thus deleted 2 of the 6 deleted subvolumes/snapshots. So I guess that part will fix itself, or I could probably delete them manually, as that is what timeshift did.

13

Yes. You can also rm -r them from the shell, but that just takes much longer.

Or delete by ID.

sudo btrfs delete -i 3294
sudo btrfs delete -i 3295

Here’s a a command I use, that will mass delete by wildcard via find:

find /mnt/sys/timeshift-btrfs/snapshots/2026-02-09_1* -maxdepth 2  -depth -name @\* -exec sudo btrfs sub delete {} \;
Delete subvolume 832 (no-commit): '/mnt/sys/timeshift-btrfs/snapshots/2026-02-09_12-02-22/@'
Delete subvolume 833 (no-commit): '/mnt/sys/timeshift-btrfs/snapshots/2026-02-09_12-02-22/@home'
Delete subvolume 834 (no-commit): '/mnt/sys/timeshift-btrfs/snapshots/2026-02-09_18-29-17/@'
Delete subvolume 835 (no-commit): '/mnt/sys/timeshift-btrfs/snapshots/2026-02-09_18-29-17/@home'

Deleting @ and @home subvolumes under: 2026-02-09_1*

Just tested it!

And you may have to use this option with your first (and maybe following deletes).

A side effect of filling up the filesystem.

            -c|--commit-after
                   wait for transaction commit at the end of the operation.
1 Like
14

I’ve removed the rest of the volumes I’ve deleted with btrfs in timeshift manually. They’ve stayed in grub, so I had to “sudo update-grub”, which cleared those up as well.

I still really don’t understand how or why this happened.
I’ve kept the volume over 15% free, there was no warning.

Based on “space_info 4” I gather that there was no metadata allocation headroom and since btrfs allocates new chunks from unallocated space, it could not allocate more and thus not balance and create more unallocated space and it all just ground to a halt?

Do I have to manually check that apart from free space I also have a lot of unallocated space? Or shoud I setup cron job with something like “sudo btrfs balance start -dusage=20 /”?

To check, I’ve also run scrub, as read-only

sudo btrfs scrub start -Brd /
Starting scrub on devid 1

Scrub device /dev/mapper/luks-fcd41acc-db81-4f2d-9e84-159b7c178fdf (id 1) done
Scrub started:    Mon May  4 20:51:02 2026
Status:           finished
Duration:         0:10:54
Total to scrub:   767.82GiB
Rate:             1.17GiB/s
Error summary:    no errors found

I do honestly think this is an error. This should not happen, my system was not bootable.

I’m grateful for the guidance that got me up and running, BIG THANKS @andreas85 and @Molski !
But I have no idea what the conclusion is?

15

Now that I had some time to digest this more.

It’s fixed!

But I think this -T option is leaving information out. These percentages are important, and to see your metadata vs data.

What do these lines say? (I can’t see them.)

e.g.

Data,RAID1: Size:1.59TiB, Used:1.59TiB (99.68%)
   /dev/sda1       1.59TiB
   /dev/sdb1       1.59TiB

Metadata,RAID1: Size:7.00GiB, Used:5.99GiB (85.60%)
   /dev/sda1       7.00GiB
   /dev/sdb1       7.00GiB

System,RAID1: Size:32.00MiB, Used:256.00KiB (0.78%)
   /dev/sda1      32.00MiB
   /dev/sdb1      32.00MiB

You ran out of allocatable metadata workspace, even though this is dynamically allocated in chunks as you need them.

Yes, leaving some space is pretty important, this may have been avoided. Sometimes 5% is fine, rare things with lots of small files may need over 30%. It depends on the workload.

In your dmesg it happened here:

global_block_rsv: size 536870912 reserved 536870912

I think the chunk layout was likely inefficient or fragmented before. Right now is says you have ~175 GB free, but you have less than 120 GB unallocated. So it’s still pretty fragmented.

Doing these device add/removes and balances most likely cleaned part of that up for us. Giving you some free space.

Just to be thorough, I would go through a scrub just to make sure everything is fine.

This can all be done on a live file system, but if you want to be cautious, do it before you boot or work with it. Note that this will take some time to complete…

sudo btrfs scrub start -Bd /mnt/sys/

If this passes I would boot normally, Then you can start cleaning it up, with a number like 75.

sudo btrfs balance start -dusage=75 -musage=75 /
2 Likes
16

Thank you very much.

Tbh, I’m still a bit uneasy about this, but even if it happens again, I can see now that this is 2 minute fix in tty and something that can be monitored for.

I have edited the title, as you’ve requested. The update was probably the last blow, but not the cause.

As I’ve run the scrub in RO mode, I know it’s under an hour, so I’ll run it again once system is backed up and then do the long balance the day after, as that will probably take a good while.

I can see your point on the formatting of the usage hides some of the information away

But based on the structured output, it was both about 90% when it broke:
Data Total 934.75
Data Used 835.33
89.36%

Metadata Total 5
Metadata Used 4.5
90%

Now it stis like this:

sudo btrfs filesystem usage  /   

Overall:
    Device size:                 944.77GiB
    Device allocated:            825.13GiB
    Device unallocated:          119.63GiB
    Device missing:                  0.00B
    Device slack:                    0.00B
    Used:                        766.53GiB
    Free (estimated):            174.13GiB      (min: 114.31GiB)
    Free (statfs, df):           174.13GiB
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:              512.00MiB      (used: 0.00B)
    Multiple profiles:                  no

Data,single: Size:815.01GiB, Used:760.51GiB (93.31%)
   /dev/mapper/luks-fcd41acc-db81-4f2d-9e84-159b7c178fdf         815.01GiB

Metadata,DUP: Size:5.00GiB, Used:3.01GiB (60.17%)
   /dev/mapper/luks-fcd41acc-db81-4f2d-9e84-159b7c178fdf          10.00GiB

System,DUP: Size:64.00MiB, Used:128.00KiB (0.20%)
   /dev/mapper/luks-fcd41acc-db81-4f2d-9e84-159b7c178fdf         128.00MiB

Unallocated:
   /dev/mapper/luks-fcd41acc-db81-4f2d-9e84-159b7c178fdf         119.63GiB

I’ll update after scrubbing and after re-balancing.

Hm, some guy on EOS having very similar issue.

For reference, I’m running:

btrfs --version                                                                                                                                                             
btrfs-progs v6.19.1
-EXPERIMENTAL -INJECT -STATIC +LZO +ZSTD +UDEV +FSVERITY +ZONED CRYPTO=libgcrypt
2 Likes
17

Maybe a bit overkill, but I installed btrfsmaintenance so I don’t have to worry about it anymore.

sudo pacman -Si btrfsmaintenance

2 Likes
18

For the record, just installing it alone doesn’t do much good. You also have to activate the individual options you want in /etc/default/btrfsmaintenance, and you should have cronie running, because although some of the options use a systemd timer, others use entries under /etc/cron.*/.

2 Likes
19

Thanks for thinking of me. My wording was unclear; English isn’t my first language.

I installed and configured it a long time ago, and all the timers are running.

3 Likes
20

The conclusion could be:

  • btrfs is a file system—like all others—that may still contain bugs.
  • btrfs is extremely robust and can withstand a great deal without losing data that has already been written (COW).
  • btrfs is quite forgiving; in most cases, even after encountering problems, the file system itself can still be considered undamaged.
  • At the same time, btrfs requires a significantly different approach compared to other file systems, and can therefore cause you a considerable amount of stress.

:footprints:

1 Like
21

This thread proves again for me: stay with ext4 … can’t recall any probs !

1 Like