BTRFS RAID10, unallocated space on disks - How to allocate?

Superkikim · 15 December 2023 10:08

Good morning/afternoon/evening,

I’m totally confused. My BTRFS, created in 2020, ran smoothly until now. I have no record of how I created it … It’s a bit old.

my problem is today it doesn’t allow me to write more data and I get why. But I could not find a way to allocate more space from physical drives to the RAID10 volume. I spent two hours this morning searching the web, and chatting with ChatGPT with no success whatsoever.

I have no subvolumes, no snapshots.

These are 8 spinning drives of 2TB each:

lsblk

sda                                             8:0    0   1.8T  0 disk  /data
sdb                                             8:16   0   1.8T  0 disk  
sdc                                             8:32   0   1.8T  0 disk  
sdd                                             8:48   0   1.8T  0 disk  
sde                                             8:64   0   1.8T  0 disk  
sdf                                             8:80   0   1.8T  0 disk  
sdg                                             8:96   0   1.8T  0 disk  
sdh                                             8:112  0   1.8T  0 disk

btrfs fi show:

Label: 'better10'  uuid: 7e504619-58c8-4f44-acbc-07a1e1ddc3f0
	Total devices 8 FS bytes used 1.32TiB
	devid    1 size 1.82TiB used 340.00GiB path /dev/sda
	devid    2 size 1.82TiB used 339.03GiB path /dev/sdb
	devid    3 size 1.82TiB used 339.00GiB path /dev/sdc
	devid    4 size 1.82TiB used 340.00GiB path /dev/sdd
	devid    5 size 1.82TiB used 339.00GiB path /dev/sde
	devid    6 size 1.82TiB used 339.03GiB path /dev/sdf
	devid    7 size 1.82TiB used 340.01GiB path /dev/sdg
	devid    8 size 1.82TiB used 340.01GiB path /dev/sdh

btrfs filesystem usage /data

Overall:
    Device size:		  14.55TiB
    Device allocated:		   2.65TiB
    Device unallocated:		  11.90TiB
    Device missing:		     0.00B
    Device slack:		     0.00B
    Used:			   2.65TiB
    Free (estimated):		   5.95TiB	(min: 5.95TiB)
    Free (statfs, df):		   5.95TiB
    Data ratio:			      2.00
    Metadata ratio:		      2.00
    Global reserve:		 512.00MiB	(used: 0.00B)
    Multiple profiles:		        no

Data,RAID10: Size:1.32TiB, Used:1.32TiB (99.89%)
   /dev/sda	 339.00GiB
   /dev/sdb	 339.00GiB
   /dev/sdc	 339.00GiB
   /dev/sdd	 339.00GiB
   /dev/sde	 339.00GiB
   /dev/sdf	 339.00GiB
   /dev/sdg	 339.00GiB
   /dev/sdh	 339.00GiB

Metadata,RAID1: Size:2.00GiB, Used:1.68GiB (84.19%)
   /dev/sda	   1.00GiB
   /dev/sdd	   1.00GiB
   /dev/sdg	   1.00GiB
   /dev/sdh	   1.00GiB

System,RAID1: Size:40.00MiB, Used:144.00KiB (0.35%)
   /dev/sdb	  32.00MiB
   /dev/sdf	  32.00MiB
   /dev/sdg	   8.00MiB
   /dev/sdh	   8.00MiB

Unallocated:
   /dev/sda	   1.49TiB
   /dev/sdb	   1.49TiB
   /dev/sdc	   1.49TiB
   /dev/sdd	   1.49TiB
   /dev/sde	   1.49TiB
   /dev/sdf	   1.49TiB
   /dev/sdg	   1.49TiB
   /dev/sdh	   1.49TiB

How to I allocate more space to my RAID10 ? (and also to the Metadata RAID1, which seems to need some) ?

I would like to add a total of 2TB to my RAID10, and 2GiB to my Metadata

I tried btrfs filesystem resize +2TB. This doesn’t work. It tries to add 2TB to /dev/sda … from 1.82TiB to 3.82TiB

Hope someone can help. I’m a bit stuck.

soundofthunder · 15 December 2023 10:13

This sounds like a job for @bedna ;
Prerequisites: 100% proof eggnog; some patience.

bedna · 15 December 2023 10:32

I think @andreas85 might be a better fit.
I have not much knowledge about btrfs + raid.

megavolt · 15 December 2023 10:48

Is it mounted ro?

mount -t btrfs

What is said in the kernel messages?

journalctl --boot=0 --dmesg --grep="btrfs"

Are the write permissions set correctly?

ls -la /data

Superkikim · 15 December 2023 12:08

Hi @megavolt

It’s mounted rw

rw,relatime,space_cache,subvolid=5,subvol=/

kernel messages are normal.

déc 13 09:57:15 cochrane kernel: Btrfs loaded, crc32c=crc32c-intel, zoned=yes, fsverity=yes
déc 13 09:57:15 cochrane kernel: BTRFS: device label better10 devid 1 transid 86808 /dev/sda scanned by (udev-worker) (769)
déc 13 09:57:15 cochrane kernel: BTRFS: device label better10 devid 2 transid 86808 /dev/sdb scanned by (udev-worker) (754)
déc 13 09:57:15 cochrane kernel: BTRFS: device label better10 devid 7 transid 86808 /dev/sdg scanned by (udev-worker) (755)
déc 13 09:57:15 cochrane kernel: BTRFS: device label better10 devid 4 transid 86808 /dev/sdd scanned by (udev-worker) (745)
déc 13 09:57:15 cochrane kernel: BTRFS: device label better10 devid 6 transid 86808 /dev/sdf scanned by (udev-worker) (775)
déc 13 09:57:15 cochrane kernel: BTRFS: device label better10 devid 3 transid 86808 /dev/sdc scanned by (udev-worker) (779)
déc 13 09:57:15 cochrane kernel: BTRFS: device label better10 devid 8 transid 86808 /dev/sdh scanned by (udev-worker) (739)
déc 13 09:57:15 cochrane kernel: BTRFS: device label better10 devid 5 transid 86808 /dev/sde scanned by (udev-worker) (758)
déc 13 09:57:16 cochrane kernel: BTRFS info (device sda): first mount of filesystem 7e504619-58c8-4f44-acbc-07a1e1ddc3f0
déc 13 09:57:16 cochrane kernel: BTRFS info (device sda): using crc32c (crc32c-intel) checksum algorithm
déc 13 09:57:16 cochrane kernel: BTRFS info (device sda): disk space caching is enabled

Not sure what is the relationship with permissions here, but here is ls -la /data

drwxrwxr-x  1 superkikim superkikim  178  8 déc 01:49 .
drwxr-xr-x 23 root       root       4096 15 déc 10:38 ..
drwxr-xr-x  1 superkikim superkikim  856  7 déc 23:26 backups
drwxr-xr-x  1 superkikim superkikim 3574 12 déc 11:37 Downloads
drwxr-xr-x  1 superkikim superkikim 8258 12 déc 11:42 InstantUpload
drwxr-xr-x  1 superkikim superkikim  266 12 déc 11:37 kdrive_silou
drwx------  1 superkikim superkikim   70 28 fév  2023 multimedia
drwxr-xr-x  1 superkikim superkikim    0 29 déc  2020 ntnxhdd
drwxr-xr-x  1 superkikim superkikim 4746 12 déc 11:41 photos
drwxr-xr-x  1 superkikim superkikim 3164  8 mai  2023 sacaidees
drwx------  1 superkikim superkikim   18  8 déc 01:49 .Trash-1000
drwxr-xr-x  1 superkikim superkikim   12  5 aoû  2020 utils

megavolt · 15 December 2023 12:15

Compare the ID numbers:

ls -lan /data
id $USER

Anything else looks normal. These have 1.49TiB unallocated on each drive . It should allocate it by itself. See:

unallocated means: No chunks were written here to store extents.

Superkikim · 15 December 2023 12:27

Tx. ID are good.

I think it’s mainly a misunderstanding on how btrfs works. That said, I don’t get why it doesn’t allocate more space:

Data,RAID10: Size:1.32TiB, Used:1.32TiB (99.89%)

I have to try a rebalance and see what is happening. I wish there were some kind of error message or warning about that. Really weird to open the system a morning and have an app telling there is no more space on the drive…

I’ll post here the result of the rebalance. In between, I have deleted 800G of obsolete data and I’m currently running a scrub. i’ll wait for it to end before I start rebalacing

andreas85 · 15 December 2023 12:38

It does so, if it is needed.
Btrfs keeps ALL Space unallocated, until it is neccesary to use it.

To have unallocated space in itself is NO problem. So where is your real problem ?

What APP ?
What is the message in detail ?
Did you rebalance after adding 7 devices to your first volume ?
( You need to rebalance at least once after creating a multidevice-setup )

Please be carefull when rebalancing.

You need to use a filter, so only those chunks are rebalanced that ARE out of balance. Otherwise you will have to wait a very long (!!!) time until all data is read and written again.

You can find good Information about Btrfs in the wiki

Superkikim:

btrfs filesystem usage /data

Overall:
    Device size:		  14.55TiB
    Device allocated:		   2.65TiB
    Device unallocated:		  11.90TiB
    Device missing:		     0.00B
    Device slack:		     0.00B
    Used:			   2.65TiB
    Free (estimated):		   5.95TiB	(min: 5.95TiB)
    Free (statfs, df):		   5.95TiB
    Data ratio:			      2.00
    Metadata ratio:		      2.00
    Global reserve:		 512.00MiB	(used: 0.00B)
    Multiple profiles:		        no

Data,RAID10: Size:1.32TiB, Used:1.32TiB (99.89%)
   /dev/sda	 339.00GiB
   /dev/sdb	 339.00GiB
   /dev/sdc	 339.00GiB
   /dev/sdd	 339.00GiB
   /dev/sde	 339.00GiB
   /dev/sdf	 339.00GiB
   /dev/sdg	 339.00GiB
   /dev/sdh	 339.00GiB

Metadata,RAID1: Size:2.00GiB, Used:1.68GiB (84.19%)
   /dev/sda	   1.00GiB
   /dev/sdd	   1.00GiB
   /dev/sdg	   1.00GiB
   /dev/sdh	   1.00GiB

System,RAID1: Size:40.00MiB, Used:144.00KiB (0.35%)
   /dev/sdb	  32.00MiB
   /dev/sdf	  32.00MiB
   /dev/sdg	   8.00MiB
   /dev/sdh	   8.00MiB

Unallocated:
   /dev/sda	   1.49TiB
   /dev/sdb	   1.49TiB
   /dev/sdc	   1.49TiB
   /dev/sdd	   1.49TiB
   /dev/sde	   1.49TiB
   /dev/sdf	   1.49TiB
   /dev/sdg	   1.49TiB
   /dev/sdh	   1.49TiB

You chose a really unusual layout for your btrfs.

There’s definitely a reason for that. Knowing that could be helpful.

Obviously you have not integrated any partitions but 8 complete disks
The data actually appears to be RAID10.
Mathematically, the total size of 8x 1.8TB with RAID10 should be 7.2TB
And btrfs also shows that only 400GB of the 1.8TB is used per disk.

Based on this information, you are definitely not out of space.

If the balance wasn’t right, at least one disk would be completely full.

Please provide real complete output ! (no edited texts) like:

mount -t btrfs
/dev/sda2 on / type btrfs (rw,relatime,lazytime,compress=zstd:9,ssd,space_cache,commit=300,subvolid=36033,subvol=/@)

Superkikim · 16 December 2023 06:40

haha no no. … It’s been a 8 drives volume since 2020. Nothing has been changed.

The app is infomaniak kdrive client on linux. I think the last release might be buggy.

andreas85 · 16 December 2023 08:00

Am I correct in assuming that you can write to your file system and only the app shows the error? (Then please mark this as solved )

And please “file a bug” with “infomaniak” and let them know that with Btrfs they:

should not pay attention to the percentage value in the line

but instead look into:

or even better into

Superkikim · 20 December 2023 08:40

Thank you for your time.

I have a ticket opened with Infomaniak. They are not very reactive ! I might change service soon …

Normally, their app is only looking at OS data, not driver data. So they should not check BTRFS usage or capacity. Linux is saying the full capacity and correct free space on the mounted FS.

Anyway, happy to understand my BTRFS has no issue

system · 20 December 2023 11:41

This topic was automatically closed 3 hours after the last reply. New replies are no longer allowed.