In another place one user threw in an udev rule which would disable write-cache for devices when they were added and while it was just an idea - it triggered my curiosity.
I dug into the intricacies of udev and found a method to only target USB storage devices.
Then plug an usb device - open in your file manager - copy a huge amount of files to the device - when the copy is done - click eject in the file manager - note how quick the device is ejected.
For those preferring the package manager, I have created a PKGBUILD which will pull the hdparm dependency upon installation.
pamac build udev-usb-sync
Another fine utility script provided by @cscs fine-tunes a number of system parameters with the option to input your own values when the script is run
Thank you for this. I had been having problems with re-purposed 2.5" SATA, non-SSD drives which I have put in USB 3.0 enclosures. I tried using the USB 3 ports on laptop versus my USB 3 hub and that made no difference. Swap settings didnât help either. Interestingly, I did not have this issue with the commercial type drives (WD My Passport, Toshiba, etc.). On the drives in the enclosure(s) it would choke with the message âQueuedâ which when I changed a Thunar setting "Transfer files in parallel, it seemed better, but would take a very long time to eject/safely remove the drive. I also tried different formatting(ext4, xfs, etc.) which made no difference.
I just implemented your solution(manually), rebooted and the process was smooth, no issues. I copied over a 15GB .vdi, unmounted/removed and remounted. All is good. This is a preliminary impression, but I feel confidant with this going forward. Thanks again.
There has been a lot of topics over time on the subject - just search the forum.
This is meant as a simple - just working - substitution for all the tweaking and fiddling with dirty_bytes and whatnot you will find when searching the forum for the subject.
Example of a solution which likely also works - please read the entire thread - to better understand what happens
I think Iâve come up with a real solution to this problem that can be evaluated to come by default in Manjaro, posted in another topic about this same problem:
Why does the solution I presented do what every place Iâve seen addressing this topic says itâs not possible to do, specifying a different cache size in ram for each device.
I used a udev rule that defines 16 MB of cache in ram for any storage device plugged into a USB port, as long as it is not the device where the / system partition is located
It doesnât change dirty pages of the system as a whole, it doesnât force sync, it just changes a configuration that is equivalent to dirty pages, but that is specific for each device.
And itâs simple to customize with other cache values or by making modifications to the udev check.
My original topic doesnât deal with dirty pages or anything else - there is a reference to a topic but that is merely reminder that we have seen these topics from time to time and they always end with various tuning suggestions which always is a matter of system and use case.
What I suggest is a simple disable write-cache for the added device if it is an usb disk.
Your suggestion is not bad but I still donât see why your suggestion is
Your repo was created March 11. - this topic is from March 4. - are you trying to promote something?
I am not critizing your suggestion, I merely curious as to why you think your solution is superior?
Does it mean that the idea presented in the OT or the idea presented by @megavolt is less real?
It is just another way of reaching the goal of having the data written and when no more data - we are done and the device can be ejected.
I donât know how you interpret the term real solution, English is not my native language, but the difference I looked for when making the solution I presented was to find a method that could be installed in the distribution by default and solve the problem in a way that is useful for almost all users.
The solutions presented before generate effects that, in my point of view, prevent them from being applied in the system.
Your idea and implementation of the idea is great.
The earlier discussions has always ended with some system tweaking which always addressed the system as a whole and thus not targeting removable USB media.
In that regard you are correct - all the previous topics did not address directly USB devices - as it should have.
This is also why I created this topic - a simple method to address the single specific usb attached storage device.
My idea - which was not mine at all - I just expanded on it to only target USB devices and using hdparm to disable write-cache.
Your idea is change the dirty_pages to a size minimal enough to flush quickly - also for the given USB device.
Both works - which one you decide to use - I donât care - I had my greatest time figuring it out - passing on the knowledge is the least of it.
Perhaps I can enhance your understanding of a solution and a real solution.
A solution could be a work-around - a monkey patch - a bandaid - name it what you like - what matters is that the solution work - but it doesnât really handle the root cause.
Compared to the above example the real solution is the one which handles the root cause.
We cannot change how the kernel works - but we can apply some rules as how different situations should be handled.
When you look at it that way - there may be multiple options available to achieve the end goal - to be able to remove the USB device immediately when the copy process ends - not waiting - perhaps several mintes before the stick can be removed.
To that end you idea and the idea I grabbed and worked with is equically good.
Hi! Thanks for your hard work and research, I really like the simplicity of this udev rule, but for some reason it seems to not be properly working for me. After creating the rule as described, I enabled logging to check to see if the rule is actually ran:
Aug 25 11:37:41 odin (udev-worker)[10420]: sdd: /etc/udev/rules.d/99-usb-sync.rules:3 RUN '[[ $(uname -r | awk -F'.' '{print $1 $2}') < 62 ]] && [[ $(which hdparm) =~ (hdparm) ]] && /usr/bin/hdparm -W 0 /dev/%k'
Aug 25 11:37:41 odin (udev-worker)[10420]: sdd: /etc/udev/rules.d/99-usb-sync.rules:9 RUN '/usr/bin/echo 1 > /sys/block/%k/bdi/strict_limit'
Aug 25 11:37:41 odin (udev-worker)[10420]: sdd: /etc/udev/rules.d/99-usb-sync.rules:9 RUN '/usr/bin/echo 50 > /sys/block/%k/bdi/max_ratio'
Aug 25 11:37:41 odin (udev-worker)[10420]: sdd: /etc/udev/rules.d/99-usb-sync.rules:9 RUN '/usr/bin/echo 16777216 > /sys/block/%k/bdi/max_bytes'
Aug 25 11:37:41 odin (udev-worker)[10420]: sdd: Setting permissions /dev/sdd, uid=0, gid=995, mode=0660
Aug 25 11:37:41 odin (udev-worker)[10420]: sdd: Successfully created symlink '/dev/disk/by-diskseq/27' to '/dev/sdd'
Aug 25 11:37:41 odin (udev-worker)[10420]: sdd: Successfully created symlink '/dev/disk/by-path/pci-0000:15:00.0-usb-0:2:1.0-scsi-0:0:0:0' to '/dev/sdd'
Aug 25 11:37:41 odin (udev-worker)[10420]: sdd: Successfully created symlink '/dev/disk/by-id/usb-SanDisk_Cruzer_20043515131DE8F0A85F-0:0' to '/dev/sdd'
Aug 25 11:37:41 odin (udev-worker)[10420]: sdd: Successfully created symlink '/dev/block/8:48' to '/dev/sdd'
Aug 25 11:37:41 odin (udev-worker)[10420]: sdd: sd-device: Created db file '/run/udev/data/b8:48' for '/devices/pci0000:00/0000:00:02.1/0000:02:00.0/0000:03:0c.0/0000:15:00.0/usb3/3-2/3-2:1.0/host12/target12:0:0/12:0:0:0/block/s>
Aug 25 11:37:41 odin (udev-worker)[10420]: sdd: Running command "[[ $(uname -r | awk -F'.' '{print $1 $2}') < 62 ]] && [[ $(which hdparm) =~ (hdparm) ]] && /usr/bin/hdparm -W 0 /dev/sdd"
Aug 25 11:37:41 odin (udev-worker)[10420]: sdd: Starting '[[ $(uname -r | awk -F'.' '{print $1 $2}') < 62 ]] && [[ $(which hdparm) =~ (hdparm) ]] && /usr/bin/hdparm -W 0 /dev/sdd'
Aug 25 11:37:41 odin (udev-worker)[10420]: Successfully forked off '(spawn)' as PID 10433.
Aug 25 11:37:41 odin (udev-worker)[10420]: sdd: Process '[[ $(uname -r | awk -F'.' '{print $1 $2}') < 62 ]] && [[ $(which hdparm) =~ (hdparm) ]] && /usr/bin/hdparm -W 0 /dev/sdd' failed with exit code 1.
Aug 25 11:37:41 odin (udev-worker)[10420]: sdd: Command "[[ $(uname -r | awk -F'.' '{print $1 $2}') < 62 ]] && [[ $(which hdparm) =~ (hdparm) ]] && /usr/bin/hdparm -W 0 /dev/sdd" returned 1 (error), ignoring.
Aug 25 11:37:41 odin (udev-worker)[10420]: sdd: Running command "/usr/bin/echo 1 > /sys/block/sdd/bdi/strict_limit"
Aug 25 11:37:41 odin (udev-worker)[10420]: sdd: Starting '/usr/bin/echo 1 > /sys/block/sdd/bdi/strict_limit'
Aug 25 11:37:41 odin (udev-worker)[10420]: Successfully forked off '(spawn)' as PID 10434.
Aug 25 11:37:41 odin (udev-worker)[10420]: sdd: '/usr/bin/echo 1 > /sys/block/sdd/bdi/strict_limit'(out) '1 > /sys/block/sdd/bdi/strict_limit'
Aug 25 11:37:41 odin (udev-worker)[10420]: sdd: Process '/usr/bin/echo 1 > /sys/block/sdd/bdi/strict_limit' succeeded.
Aug 25 11:37:41 odin (udev-worker)[10420]: sdd: Running command "/usr/bin/echo 50 > /sys/block/sdd/bdi/max_ratio"
Aug 25 11:37:41 odin (udev-worker)[10420]: sdd: Starting '/usr/bin/echo 50 > /sys/block/sdd/bdi/max_ratio'
Aug 25 11:37:41 odin (udev-worker)[10420]: Successfully forked off '(spawn)' as PID 10435.
Aug 25 11:37:41 odin (udev-worker)[10420]: sdd: '/usr/bin/echo 50 > /sys/block/sdd/bdi/max_ratio'(out) '50 > /sys/block/sdd/bdi/max_ratio'
Aug 25 11:37:41 odin (udev-worker)[10420]: sdd: Process '/usr/bin/echo 50 > /sys/block/sdd/bdi/max_ratio' succeeded.
Aug 25 11:37:41 odin (udev-worker)[10420]: sdd: Running command "/usr/bin/echo 16777216 > /sys/block/sdd/bdi/max_bytes"
Aug 25 11:37:41 odin (udev-worker)[10420]: sdd: Starting '/usr/bin/echo 16777216 > /sys/block/sdd/bdi/max_bytes'
Aug 25 11:37:41 odin (udev-worker)[10420]: Successfully forked off '(spawn)' as PID 10436.
Aug 25 11:37:41 odin (udev-worker)[10420]: sdd: '/usr/bin/echo 16777216 > /sys/block/sdd/bdi/max_bytes'(out) '16777216 > /sys/block/sdd/bdi/max_bytes'
Aug 25 11:37:41 odin (udev-worker)[10420]: sdd: Process '/usr/bin/echo 16777216 > /sys/block/sdd/bdi/max_bytes' succeeded.
Aug 25 11:37:41 odin (udev-worker)[10420]: sdd: Adding watch on '/dev/sdd'
/etc/udev/rules.d ⯠cat /sys/block/sdd/bdi/strict_limit
0
/etc/udev/rules.d ⯠cat /sys/block/sdd/bdi/max_ratio
100
/etc/udev/rules.d ⯠cat /sys/block/sdd/bdi/max_bytes
4462813184
/etc/udev/rules.d ⯠inxi
CPU: 8-core AMD Ryzen 7 7700X (-MT MCP-) speed/min/max: 3210/3000/5572 MHz
Kernel: 6.4.9-1-MANJARO x86_64 Up: 1h 6m Mem: 10.04/31.06 GiB (32.3%)
Storage: 13.65 TiB (27.7% used) Procs: 416 Shell: Zsh inxi: 3.3.29
It seems the hdparm command fails for some reason, and I do have hdparm installed. But the second set of commands to write to /sys/block seems to succeed, but when I read those files after the values are not written to them. I am on kernel 6.4 so Iâd imagine that part of the rule should work. Any idea whatâs going wrong here?
I have recently modified the rule for hdparm as I wanted to make sure the rule was only executed when hdparm is actually installed and the kernel is < than 62
This is why the hdparm doesnât execute - the modification prevent it if kernel is > 6.1
If you want to ensure the rule executes change from
Well either way, shouldnât the second portion of the rule still be able to function? Any idea of whatâs keeping it from successfully writing to /sys/block ?
Changing the rule to what you described and reloading the rules still leaves the USB devices with very long removal times.
Disabling write cache and decreasing the buffer size does not increase the transfer speed to the device - it only ensures that when the copy is done there is no buffer left to flush.
Ah, but by long removal times I mean itâs reporting the copy as âdoneâ (probably due to misrepresenting the transfer speed) but will not let me safely remove (or eject) the device for sometimes upwards of 10 minutes, depending on the speed of the device or the size of the file(s). I have no intent on trying to speed up the device, I would just like the file transfer to be reported correctly and be able to eject the device after the copy is reported as complete.
