Slow writes with Btrfs + LUKS + NVME SSD

[Tobiwan]

Hi,

my problem: Writes on my NVME-SSD are very slow.
second problem (I think it is connected with my first): Running btrfs balance regularly locks up my computer for multiple seconds - programs like the KDE UI, Firefox and Thunderbird regularly crashes when I try to interact with them during a rebalance.

I thought that I am an experienced Linux user but I couldn’t find a solution (Btrfs slow read/write speed on nvme did not help me)

My suspicion: Maybe I did something wrong when I extended the BTRFS partition after migration to a 2TB and later to a 4TB SSD.

I’m also using snapshots (Timeshift create a new snapshot each boot) ans quotas (for size information in Timeshift).

Read/Write performance:

The performance is similar on kernel 6.1, 6.2 and 6.3.

/etc/fstab:

# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a device; this may
# be used with UUID= as a more robust way to name devices that works even if
# disks are added and removed. See fstab(5).
#
# <file system>             <mount point>  <type>  <options>  <dump>  <pass>
UUID=B253-C727                            /boot/efi      vfat    umask=0077 0 2
/dev/mapper/luks-205e4e87-163d-4dae-bd7c-3cda9971fa7f /              btrfs   subvol=/@,defaults,noatime,discard=async,ssd,clear_cache,nospace_cache 0 0
/dev/mapper/luks-205e4e87-163d-4dae-bd7c-3cda9971fa7f /home          btrfs   subvol=/@home,defaults,noatime,discard=async,ssd,clear_cache,nospace_cache  0 0
/dev/mapper/luks-205e4e87-163d-4dae-bd7c-3cda9971fa7f /var/cache     btrfs   subvol=/@cache,defaults,noatime,discard=async,ssd,clear_cache,nospace_cache  0 0
/dev/mapper/luks-205e4e87-163d-4dae-bd7c-3cda9971fa7f /var/log       btrfs   subvol=/@log,defaults,noatime,discard=async,ssd,clear_cache,nospace_cache  0 0
/dev/mapper/luks-8b452496-c619-4399-b0f5-0e387d52567b swap           swap    defaults,noatime,sdd 0 0
tmpfs                                     /tmp           tmpfs   defaults,noatime,mode=1777 0 0
/dev/disk/by-id/wwn-0x50014ee26a3cac5a-part1 /mnt/backup_harddrive_green_1 auto nosuid,nodev,nofail,noatime,noauto,x-gvfs-show 0 0
/dev/disk/by-id/wwn-0x50014ee211edd557-part1 /mnt/backup_harddrive_red_1 auto nosuid,nodev,nofail,noatime,noauto,x-gvfs-show 0 0
/dev/disk/by-id/wwn-0x50014ee2bc98dfb8-part1 /run/media/hacker/BackupBlau/ auto nosuid,nodev,nofail,noatime,noauto,x-gvfs-show 0 0
/dev/disk/by-id/wwn-0x50014ee215b34320-part1 /mnt/backup_violett auto nosuid,nodev,nofail,noauto,noatime,x-gvfs-show 0 0
/dev/disk/by-id/wwn-0x50014ee2c0719f7e-part1 /mnt/backup_tuerkis auto nosuid,nodev,nofail,noauto,noatime,x-gvfs-show 0 0
/dev/disk/by-uuid/a2766337-56fd-44b7-a0d6-df8521308e79 /mnt/linux_games auto nosuid,nodev,nofail,noatime,x-gvfs-show 0 0

I also enabled fstrim:

sudo systemctl enable --now fstrim.timer && sudo systemctl start fstrim

btrfs fi us / :

WARNING: cannot read detailed chunk info, per-device usage will not be shown, run as root
Overall:
    Device size:                   3.61TiB
    Device allocated:              1.88TiB
    Device unallocated:            1.73TiB
    Device missing:                  0.00B
    Device slack:                 16.00EiB
    Used:                          1.84TiB
    Free (estimated):              1.76TiB      (min: 920.58GiB)
    Free (statfs, df):             1.76TiB
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:              512.00MiB      (used: 0.00B)
    Multiple profiles:                  no

Data,single: Size:1.87TiB, Used:1.83TiB (98.12%)

Metadata,DUP: Size:7.00GiB, Used:5.91GiB (84.48%)

System,DUP: Size:32.00MiB, Used:288.00KiB (0.88%)

sudo btrfs device stats /                                                                                                                        PIPE|INT ✘  3s  
[/dev/mapper/luks-205e4e87-163d-4dae-bd7c-3cda9971fa7f].write_io_errs    0
[/dev/mapper/luks-205e4e87-163d-4dae-bd7c-3cda9971fa7f].read_io_errs     0
[/dev/mapper/luks-205e4e87-163d-4dae-bd7c-3cda9971fa7f].flush_io_errs    0
[/dev/mapper/luks-205e4e87-163d-4dae-bd7c-3cda9971fa7f].corruption_errs  522
[/dev/mapper/luks-205e4e87-163d-4dae-bd7c-3cda9971fa7f].generation_errs  0

[andreas85]

You always should run btrfs filesystem usage as root. Otherwise the information is incomplete (or partially wrong)

Please only balance btrfs if it is really necessary if you are on an encrypted system.

It is necessary

• if the RAID level was changed
• Or when the partition was almost full.

@Tobiwan

Device allocated: 1.88TiB

→ balance is not neccesary

As long as btrfs shows enough unallocated space, a balance is unnecessary and only puts a strain on the SSD and the system.

With an unencrypted system, a balance will only read and rewrite the data. With an encrypted system, however, they are decrypted, read, rewritten and also re-encrypted. Of course, this puts a strain on the CPU.

---

You can find good Information about Btrfs in the wiki

Btrfs

[TriMoon]

@andreas85

As long as btrfs shows enough unallocated space, a balance is unnecessary and only puts a strain on the SSD and the system.

With an unencrypted system, a balance will only read and rewrite the data. With an encrypted system, however, they are decrypted, read, rewritten and also re-encrypted. Of course, this puts a strain on the CPU.

And on top of that:

@Tobiwan

I’m also using snapshots (Timeshift create a new snapshot each boot) ans quotas (for size information in Timeshift).

If i’m not mistaken, a balance also breaks the connection between the snapshot and original files which leads to more data use needed in the partition.

Thus in combination with an encrypted volume that has BTRFS inside leads to extra strain on your CPU and SSD like @andreas85 already explained…

[megavolt]

Hello @Tobiwan

As already said: encryption degrades writing throughput, but also nospace_cache leads to higher latency, which means: the driver has to scan the whole file system until it finds free space, the cache improves this drastically, but the downside is a possible reduce of life span of the nvme. When it scans, it has to decrypt the data every time if not cached in RAM.

[andreas85]

@TriMoon

If i’m not mistaken, a balance also breaks the connection between the snapshot and original files which leads to more data use needed in the partition.

It will not break the connection between snapshots.

But if some parts of files have been deduplicated, this may break off, so the used space may be a little bigger afterwards. But it also may deduplicate other parts, and gain some space. This may be a game with some win and some loss.

You may gain some speed if compression is enabled

[Tobiwan]

Thanks for your help.

• I was not aware that Manjaro has its own wiki. I only knew Arch Wiki.
• I only read once that Balance should not be executed regularly. I though that this info is outdated
• I will remove “nospace_cache” and check if this helps

[andreas85]

@Tobiwan

I only read once that Balance should not be executed regularly. I though that this info is outdated

It depends

on the filters you use

A full balance (no filter used) is cost-full
A little balance (filter on <=50%) is no problem

[HowTo] get Space on BTRFS

If you encounter a situation where a Btrfs volume is out of space, There is a way out However, you have to accept that Btrfs is a “strange” filesystem. So if you’re doing things that would help with other filesystems, it might NOT help with btrfs. If you want to know why, check out the wiki Here are a few simple tips: 1. Only use btrfs to meassure the unallocated space on your filesystem: sudo btrfs filesystem usage / 2. Look at “Device unallocated:” If the Unallocated size is less than 10GB…

[Tobiwan]

Current situation: The first GBs are written really fast. But after that I can barely make 100MB/s.

It seems to be that my Crucial P3 4TB is the problem. Reddit - Dive into anything

I issued a TRIM command with sudo fstrim -Av and I will check tomorrow if this solves the problem. (I know that this kind of SSD has only a small and fast write cache - but 10GB on a 4TB drive is a joke)

EDIT:
Good to know - TRIM does not work on LUKS out of the box dm-crypt/Specialties - ArchWiki

[Tobiwan]

Seems that I fixed the issue by allowing TRIM on Luks

(by adding :allow-discards to cryptdevice=UUID=205e4e87-163d-...:luks-... in GRUB_CMDLINE_LINUX_DEFAULT= in /etc/default/grub + sudo update-grub)

[andreas85]

@Tobiwan

I’m also using snapshots (Timeshift create a new snapshot each boot) ans quotas (for size information in Timeshift).

Quotas are a fine feature, but:

• They are known to cause problems with snapshots (especially with many snapshots) - This appears to have been corrected. The corresponding entry in the Btrfs documents has disappeared.
• will slow down the release of snapshots
• will REALY slow down balance

→ see the btrfs-wiki at : btrfs.readthedocs.io

I had a system, where quotas have been enabled. After that a balance could take hours, or a whole day. Since quotas are disabled, the system has no problems any more (This has been years ago, but i would not like to test this again)