What level of compression is advantageous for BTRFS?

Support Software & Applications
1

The existing text in the wiki isn’t very well-founded.

You can turn on compression at any time by changing the mount options in the fstab. After the next reboot, all newly written data from then on will be automatically compressed. Compression factors of 2:1 on average are not uncommon. @see mount-options

UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx / btrfs rw,noatime,compress=zstd,subvol=@ 0 0

Which compression suits me best?

CPU rotating disk SSD
slow zstd:1 … 4 zstd:1 … 3
medium zstd:4 … 7 zstd:3 … 5
fast zstd:7 … 10 zstd:5 … 10

Compression can only be set per volume. So if you set different “levels” of compression for different mounts in the fstab, only the first “level” will be applied. see@btrfs.readthedocs.io

Perhaps someone has had their own experiences so that the wiki could be generalized and improved. @Molski @CGA @linux-aarhus

Why I’ve been using zstd:9 so far. I assumed:

  • most file operations are reads.
    • btrfs-zstd utilizes multiple cores(?). My system has 8 cores and 16 threads.
    • with zstd, the time required to decompress reads hardly depends on the compression factor.
    • with better compression, the read yields more data per decompressed block.
  • most of the data I write is rather small.
    • Compression also uses multiple threads(?) and benefits from my processor.
    • A slow hard drive (like the one I had back then) benefits more from good compression because it definitely reads/writes data slower than zstd can process it.
  • New arguments include:
    • I now also have an NVMe drive (in RAID with a SATA SSD)
    • The really large Manjaro updates can write several GB in a short time
    • Generating the initrd also writes a lot of data at once

:footprints:

Tidied markdown to allow table to render. No charge.

2

I am no good reference - I don’t use btrfs - not entirely true - but I have no experience with configuration or maintenance of btrfs.

2 Likes
3

I was actually really curious. But when I read zstd:9, I my head honestly went back to the old (and almost every other UNIX compression level that uses the scale of 1-9.) And I completely forgot btrfs went from -15 to 15.

So essentially, I thought you were running compression level zstd:15. On this “logarithmic” curve of the ratio between computation to size reduction, I though you picked the very end. (Whoops!)

I have read nothing but good things about the efficiency of compression of zstd, then combined that each extent gets a thread, makes this even more efficient. So you’re setup makes a lot more sense with a good CPU (and many logical cores). (When you actually understand the levels!)

I know new Manjaro btrfs installations used to come with the default compression level of 3. (I don’t think it does anymore, or does it?)

This is went I started testing on it (now years ago).

Much of my work, and play, involves moving and operating on large files. All the tests I did were not for parallel OR random IO, so they were completely biased, and don’t fit most use cases.

So I primary focused on single operation consecutive IO. This is where I was able to pin a (Xen 3) core, and have the operation take substantially longer. I am almost certain (at least back then), that I did not understand changing compression/CoW mount options or volume/file properties did not change existing data. (I mean, to be fair, everything else seemed to happen magically the background!)

I get, and I do make exceptions where I can, whole volumes obviously are preferred. Easy for VMs and the like, but then you have situations like haves a ton of uncompressed video or audio, sitting near it’s compressed counterparts.

So knowing what I know now, I definitely need to revisit those tests. And I will! (Today may be hard, within a day or two). And give it some more real life benchmarks.

I have a feeling I may want to revisit putting zstd:[0-5] on my @ and home (then make exceptions from there), especially with a lot of snapshot retention, it does make a lot more sense.

I think once I tried, but there was a loss of communication. And my life comes and goes in spurts on what on I can contribute.

1 Like
4

I have just recently upped the compression level from zstd 3 to 9 on my second drive which is an old 5200 rpm internal spinning rust. So far I haven’t noticed any slowdowns due to this but it’s a drive which I seldomly write to manually, mostly used for background backups.

1 Like