The main problem is related to matching the rate at which a process creates dirty memory to the rate at which that memory can be written to the underlying storage device. If a process is allowed to dirty a large amount of memory, the kernel will find itself committed to writing a chunk of data that might take minutes to transfer to persistent storage. All that data clogs up the I/O queues, possibly delaying other operations. And, as soon as somebody calls sync(), things stop until that entire queue is written. It’s a storage equivalent to the bufferbloat problem. - The pernicious USB-stick stall problem [LWN.net]
This is probably one of the oldest open issues out there with linux. I just tried to copy an mp4 file (1.8GB) to two USB flash drives: one was formatted as exFAT, the other as FAT32 and both copies failed to equal the original file from my SSD.
1st USB drive) FAT32
Hanged on “100% completed” for more than 10min, I can see the size of 1.8GB had been occupied on USB stick and when
cp process has finished, the file has suddenly disappeared from USB as it was not copied at all. Forcing
sync may have helped with this one…
2nd USB drive) exFAT
I could copy the file on USB with an expected speed for SSD to USB 3.1 data transfer, but the checksum was different, the header of the copy of the mp4 file was different and couldn’t be played on any device, even if I copy it back to the original location. The header of mp4 was significantly changed.
All this happened without any warning, so I find it as an extremely alerting sign to be careful when using USB flash drives. Since almost 8 years ago, even Linus was addressing this issue, and there’s an ongoing discussion until today.
I understand it’s due to setting the following variables:
$ sysctl -a | grep dirty vm.dirty_background_bytes = 0 vm.dirty_background_ratio = 10 vm.dirty_bytes = 0 vm.dirty_expire_centisecs = 1500 vm.dirty_ratio = 20 vm.dirty_writeback_centisecs = 1500 vm.dirtytime_expire_seconds = 43200
I’ve found somewhat outdated discussions i.e.: USB hard drive transfer forces system into a cawl / Kernel & Hardware / Arch Linux Forums
Are there any recommended values to set those defaults (for 16GB of RAM) to avoid similar issues in the future (for transferring large files and many smaller files as well)? I would assume with this kernel, it may be impossible to satisfy all scenarios out there, but maybe I’m missing something… I would like to avoid those huge hickups like described above.