Slow USB transfers

Support
7

(Sorry I can’t reply to more than 2 at once because I’m new, so I’m cutting you out Xephon! :slightly_smiling_face:)
At Xephon: Sadly, you may be correct. :face_with_diagonal_mouth:

@linux-aarhus Not to be rude, but please reread my post. It is not the cache/buffer. That’s not what I am talking about. I am aware that when dolphin says my transfer speeds are above 100 MB/s that it is merely the buffer filling up. But the emptying of that buffer should be on par with the write speed of the device, which it is not. Compare it to windows, which sustains ~80-100MB/s continuously and is done in a little over an hour.

I double checked your search link, the post that is a few days old here

(I removed link because I cant post it, I’m too new here.)

actually reinforces my point. This is the same issue as mine. 38+ hours for a 274GB transfer at ~5MB/s is insane for USB 3.0, or even 2.0. I mentioned in the OP that on windows it takes maybe an hour and 15 minutes. That is a far cry from 1 and a half days. This must be a bug of some sort. It is also mentioned on Ubuntu and Fedora forums dating back to mid 2000’s which is when I first experienced it.

I read your response here and on that page as well, pointing to modifying the udev rules. Is modifying the udev rules to disable the write-cache the same as shrinking or increasing dirty bytes? I will try it if it might help and will post the results. :slightly_smiling_face: Is there a way to do it without rebooting? I am on a Live ISO.

@megavolt: I installed iotop and ran sudo iotop -Po (-x errored out as an unavailable command) I did not use dd yet since it is still transferring. I figured it may as well be the same anyway. But I will run dd if you need a comparison of some sort.

Here is the current result and speeds. rsync has been running since about 2 AM here, so 9 hours. Rsync confirms this. It has only transferred 43 GB. When i first got home, it was bouncing between 250-700 KB/s. It’s running around 1.3MB/s at the moment.
(sigh…screenshot removed because I cant post media yet)
here is the copy and pasted outputs of 2 terminals…

iotop:

Total DISK READ :    1235.78 K/s | Total DISK WRITE :     813.52 K/s
Actual DISK READ:    1235.78 K/s | Actual DISK WRITE:       0.00 B/s
    PID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND                            
   2310 be/4 root     1235.78 K/s    0.00 B/s  ?unavailable?  mount.ntfs /dev/sd~mes,uhelper=udisks2
   5000 be/4 root        0.00 B/s  813.52 K/s  ?unavailable?  mount.ntfs /dev/sd~mes,uhelper=udisks2

rsync:

Users/USERNAME/Desktop/New folder/2017-12-01 iphone photos up until 11-30-17/iphone photos up until 11-30-17 12224.JPG
 42,766,260,000  24%    1.18MB/s    9:35:39 (xfr#43730, ir-chk=12272/165340)
Users/USERNAME/Desktop/New folder/2017-12-01 iphone photos up until 11-30-17/iphone photos up until 11-30-17 12225.MOV
 42,795,718,432  24%  708.32kB/s   52:31:19

I do not know how to use quotes here yet.
At megavolt said:

So you mean then NTFS? It isn’t a surprise to me since NTFS works on user space (ntfs-3g). It is in general slower, same as ext4 driver on Windows.

The drive im transferring from is NTFS, yes. It is an internal SATA drive.
I am transferring to either NTFS external drive over USB 3.0 (internal drive itself is SATA)
OR to a USB 3.0 stick that is exFAT.

I would blame ntfs-3g, but that doesn’t explain the perfect speeds and huge data transfer during the first 30 minutes (up to 70GB.) After which it slows to a crawl.

That is an indicator for an SMR HDD, but can be wrong. CMR HDDs, don’t have that issue.

Again, It also happens with the USB flash drive. Anywhere from 10-70GB at ~80-100MB/s speeds, then a drop to <5MB/s. I believe this plus the fact that windows has no problem at all in either case would rule out SMR.

EDIT: ~80-100MB/s were for the external HDD, not the USB flash drive which is closer to probably 45MB/s on windows

Also cheap fake USB Flash-Drives, which reports more storage, then actually there can be the issue here. It is known that those devices starts fast and when they reach a limit then they transfer in slow speed or even nothing or produce corrupted files after that.

It was a cheap drive (Got it on sale :wink:) and I have bought fake ones before. I had a 128 GB drive that was actually a 16GB flash drive that would “wrap around” and overwrite the beginning, causing some lovely corruption for family photos. This current drive is legit.

Thanks again for the replies everyone. As I’ve said, both the flash and external drive work fine under windows at the proper speeds without any corruption.

Edit: lol my quotes worked :slight_smile:

8

I don’t consider you rude - if you are 100% certain it is not cache related, and not hardware related - then I have no idea.

The thing is - I cannot reproduce such issue - and a lot of things can be done with configuration - but the thing is - there is no one-size-fits-all solution … what works for me might not work for you - however the options presented in one of the linked topics has proved to be working for some.

9

If you believe that, then I have no idea here. I don’t see such an issue with my drives and I cannot reproduce it. It works as intended.

Maybe check if the kernel has detected a zoned HDD:

 cat /sys/block/sd[a-z]/queue/zoned

Maybe here you get better information about SMR: Getting Started with SMR Hard Disks | Zoned Storage

At this stage only f2fs or btrfs have optimization for SMR on linux. No NTFS or exFAT.

I can imagine that a USB3 flash drive could get hot and heat can also throttle speed. :man_shrugging:

If possible try linux filesystem: ext4, btrfs, xfs etc.

However… I strongly believe that this is not an usb driver issue at all on linux. Maybe your mainboard has a special USB3 mode, who knows? Like USB 3.0 Boost or Turbo Mode or UASP Mode ? Maybe that comes into play here? Just puzzling.

NTFS and EXFAT are both user space by default.

10

I can confirm what @megavolt and @linux-aarhus stated, I have never seen any nameable performance differences between Windoze 7, 10 and Linux (Archlinux, Manjaro, EndeavourOS) with any type of USB stick or any external HDD, either using USB 2.0, 3.1 or 3.2 as long as NTFS is not the target filesystem with the known limitations of ntfs-3g.

I’m pretty sure your issues have something to do with your specific environment and settings. So, @zippydippy, @Xephon, please stop claiming there would be a general issue with USB performance in Linux if you cannot prove it. :face_with_raised_eyebrow:

11

So I tried doing the same thing on EndeavourOS, different machine. Ryzen 1800x, 16GB ram, 2tb Mechanical SATA HDD, USB 3.0 ports (everything else removed so no chance of 2.0 speeds) and the same thing happens using a completely different ext4 source drive with large files. Fast, followed by a drop to ~1MB/s.

I tested it again on a laptop of mine with 3.0 ports. Also formatted with ext4, and transferring to the NTFS drive or USB stick. Same results: ~1MB/s

@megavolt

Maybe check if the kernel has detected a zoned HDD:
Output:
cat /sys/block/sd[a-z]/queue/zoned  :heavy_check_mark:
none
none
none
none
none

Changing filesystems will not help in my case, I need to keep the external HDD as NTFS. The USB stick is exFAT, so no SMR, but I may reformat it to ext4 to see if it makes a difference after I modify udev next. Since both drives drop speeds, I’m not completely ruling out it being a filesystem issue.

I strongly believe that this is not an usb driver issue at all on linux.

I believe it is either a usb driver bug or kernel bug. I have consistently retested this across 3 machines now, including EndeavourOS.

@Wollie

Hi :slight_smile:

I can confirm what megavolt and linux-aarhus stated, I have never seen any nameable performance differences between Windoze 7, 10 and Linux (Archlinux, Manjaro, EndeavourOS) with any type of USB stick or any external HDD, either using USB 2.0, 3.1 or 3.2 as long as NTFS is not the target filesystem with the known limitations of ntfs-3g.

Please, anyone here, try this. Try it on your current OS, or a Live one, copy a Giant Folder (above 100GB) from an internal drive to an external 3.0 drive and post your results. Use Dolphin, cp, rsync, whatever works for you. Because the more I read this the less I am actually believing it. It is fast, but it will drop off to less than 10% theoretical speeds after some unknown amount which means it is indeed a bug of some sort.

I’m pretty sure your issues have something to do with your specific environment and settings. So, zippydippy, Xephon, please stop claiming there would be a general issue with USB performance in Linux if you cannot prove it. :face_with_raised_eyebrow:

This is the reason a lot of would be linux users are driven away. I have an issue and it is dismissed as if the situation is not real and reproducible. Granted, it may not be reproducible on certain hardware. However, I have proved it across 3 machines and two arch Linux distros, with experience of it happening in the past using both Kubuntu and Fedora using 2 other machines (modern at the time) with many posts online reiterating this experience. None of these hardware are similar and the proof has definitely been provided.

I keep reading online that there is no proof, when there clearly is in the case of slow USB transfers and never in the “it works for me” case. My claim stands, it is a real phenomenon probably coming from the kernel or USB drivers.

Please try it for yourself and post results. Copy a giant folder and post the speeds and the experience all the way to the finish line. Using an external flash/HDD is very commonplace and this is common enough situation for a bug report to be warranted.

Btw, I am still transferring that folder using rsync on the first pc. :rofl: it is at 167GB out of 262GB and has taken over 22 hours with a current rate of 300KB/s. This is the same folder that took about an hour and 15 min on windows.

linux-aarhus I will attempt the udev fix next to completely rule out the cache issue. Then I will try reformatting the USB to ext4

Sorry for the long post, please stick with me.

edited for typos

12

Even this is no scientific test - the results speaks for themselves.

You can draw your own conclusions - to me it is evident the LInux kernel is not the culprit …

 $ inxi -SC
System:
  Host: tiger Kernel: 6.5.0-rc7-next-20230824-1-next-git-12477-g2b3bd393093b
    arch: x86_64 bits: 64 Desktop: KDE Plasma v: 5.27.7 Distro: Manjaro Linux
CPU:
  Info: 12-core model: AMD Ryzen Threadripper PRO 5945WX s bits: 64
    type: MT MCP cache: L2: 6 MiB
  Speed (MHz): avg: 2269 min/max: 1800/7015 cores: 1: 1800 2: 1800 3: 1800
    4: 1800 5: 1800 6: 2940 7: 1739 8: 1800 9: 1800 10: 4100 11: 1800 12: 2940
    13: 1800 14: 1740 15: 1800 16: 1800 17: 1800 18: 4100 19: 1800 20: 1756
    21: 1757 22: 4100 23: 1800 24: 4100

Depending on your system and on the source - e.g. another disk device - reproducing will be different as these tests does not read another disk device but a stream of random bytes produced by the system. The reason for using thr random stream is simply to avoid identical streams when testing the various devices.

Reproduce the tests

Ensure your system is fully up-to-date and hdparm installed

sudo pacman -Syu hdparm

Create this udev rule (referenced earlier)

 $ cat /etc/udev/rules.d/99-usb-sync.rules 
# rule to disable write cache for usb storage
# requires hdparm to be installed
ACTION=="add|change", KERNEL=="sd[a-z]", ENV{ID_USB_TYPE}=="disk", RUN+="/usr/bin/hdparm -W 0 /dev/%k"

#
# the following rules is introduced with kernel 6.2
# https://docs.kernel.org/admin-guide/abi-testing.html#abi-sys-class-bdi-bdi-strict-limit
# https://docs.kernel.org/admin-guide/abi-testing.html#abi-sys-class-bdi-bdi-max-ratio
# https://docs.kernel.org/admin-guide/abi-testing.html#abi-sys-class-bdi-bdi-max-bytes
ACTION=="add|change", KERNEL=="sd[a-z]", ENV{ID_USB_TYPE}=="disk", RUN+="/usr/bin/echo 1 > /sys/block/%k/bdi/strict_limit",  RUN+="/usr/bin/echo 50 > /sys/block/%k/bdi/max_ratio", RUN+="/usr/bin/echo 16777216 > /sys/block/%k/bdi/max_bytes"

Reload the rules

sudo udevadm control --reload

Create a test script - requires the package time to be installed.

 $ cat test.sh
#!/usr/bin/env bash
echo "Executing: time dd if=/dev/urandom of=/mnt/test.img bs=1G count=100 status=progress"
time sudo dd if=/dev/urandom of=/mnt/test.img bs=1G count=100 status=progress
echo "Executing: time umount /mnt"
time sudo umount /mnt

You can create variations over this to simulate other conditions e.g.

# create 100 files of 100M each
for i in {1..100}; do
    sudo dd if=/dev/urandom of="/mnt/test${i}.img" bs=10M count=10
done

Mount the test device/partition on /mnt and execute …

sudo mount /dev/sdxY /mnt

SATA enclosure test

OCZ Agility 3 240G SATA in 1.gen USB 3 enclosure (? 2010)

formatted with ext4

 $ sudo bash test.sh
Executing: time dd if=/dev/urandom of=/mnt/test.img bs=1G count=100 status=progress
107374182400 bytes (107 GB, 100 GiB) copied, 664 s, 162 MB/s
100+0 records in
100+0 records out
107374182400 bytes (107 GB, 100 GiB) copied, 735,558 s, 146 MB/s

real    12m15,586s
user    0m0,005s
sys     0m0,006s
Executing: time umount /mnt

real    0m9,505s
user    0m0,004s
sys     0m0,008s

formatted with ntfs

 $ sudo bash test.sh
Executing: time dd if=/dev/urandom of=/mnt/test.img bs=1G count=100 status=progress
107374182400 bytes (107 GB, 100 GiB) copied, 676 s, 159 MB/s
100+0 records in
100+0 records out
107374182400 bytes (107 GB, 100 GiB) copied, 675,717 s, 159 MB/s

real    11m15,742s
user    0m0,011s
sys     0m0,000s
Executing: time umount /mnt

real    0m59,460s
user    0m0,004s
sys     0m0,007s

USB-C device test

Samsung 1.8T T7 Shield USB-C (july 2023)

formatted with ntfs

 $ sudo bash test.sh
Executing: time dd if=/dev/urandom of=/mnt/test.img bs=1G count=100 status=progress
107374182400 bytes (107 GB, 100 GiB) copied, 485 s, 221 MB/s
100+0 records in
100+0 records out
107374182400 bytes (107 GB, 100 GiB) copied, 485,176 s, 221 MB/s

real    8m5,205s
user    0m0,004s
sys     0m0,007s
Executing: time umount /mnt

real    0m15,391s
user    0m0,002s
sys     0m0,007s

This device proved a 3-4 minute improvement in transfer speed compared to the old SATA device

WD Passport 512G MILAN II USB-C (? 2021)

formatted with ext4

 $ subo bash test.sh
Executing: time dd if=/dev/urandom of=/mnt/test.img bs=1G count=100 status=progress
107374182400 bytes (107 GB, 100 GiB) copied, 619 s, 174 MB/s
100+0 records in
100+0 records out
107374182400 bytes (107 GB, 100 GiB) copied, 685,141 s, 157 MB/s

real    11m25,165s
user    0m0,012s
sys     0m0,000s
Executing: time umount /mnt

real    0m6,726s
user    0m0,010s
sys     0m0,000s

64G USB stick test

SanDisk Extreme 64G USB 3.0 (? 2020)

This is tested like the others but using a 50G stream

 $ cat test-50.sh
#!/usr/bin/env bash
echo "Executing: time dd if=/dev/urandom of=/mnt/test.img bs=1G count=100 status=progress"
time sudo dd if=/dev/urandom of=/mnt/test.img bs=1G count=100 status=progress
echo "Executing: time umount /mnt"
time sudo umount /mnt

formatted with exfat

 $ sudo bash test-50.sh
Executing: time dd if=/dev/urandom of=/mnt/test.img bs=1G count=50 status=progress
53687091200 bytes (54 GB, 50 GiB) copied, 869 s, 61,8 MB/s
50+0 records in
50+0 records out
53687091200 bytes (54 GB, 50 GiB) copied, 868,694 s, 61,8 MB/s

real    14m28,720s
user    0m0,004s
sys     0m0,004s
Executing: time umount /mnt

real    2m19,978s
user    0m0,001s
sys     0m0,005s

formatted with ntfs

 $ sudo bash test-50.sh
Executing: time dd if=/dev/urandom of=/mnt/test.img bs=1G count=50 status=progress
53687091200 bytes (54 GB, 50 GiB) copied, 740 s, 72,5 MB/s
50+0 records in
50+0 records out
53687091200 bytes (54 GB, 50 GiB) copied, 740,327 s, 72,5 MB/s

real    12m20,350s
user    0m0,002s
sys     0m0,006s
Executing: time umount /mnt

real    3m11,227s
user    0m0,006s
sys     0m0,000s

formatted using f2fs

 $ sudo bash test-50.sh
Executing: time dd if=/dev/urandom of=/mnt/test.img bs=1G count=50 status=progress
53687091200 bytes (54 GB, 50 GiB) copied, 701 s, 76,6 MB/s
50+0 records in
50+0 records out
53687091200 bytes (54 GB, 50 GiB) copied, 700,81 s, 76,6 MB/s

real    11m40,822s
user    0m0,001s
sys     0m0,008s
Executing: time umount /mnt

real    2m35,335s
user    0m0,000s
sys     0m0,006s

Kingston DataTraveler100 64G USB 3.1 (? 2021)

This device had been tested to showcase vendor quality

formatted using ntfs

 $ sudo bash test-50.sh
Executing: time dd if=/dev/urandom of=/mnt/test.img bs=1G count=50 status=progress
53687091200 bytes (54 GB, 50 GiB) copied, 1104 s, 48,6 MB/s
50+0 records in
50+0 records out
53687091200 bytes (54 GB, 50 GiB) copied, 1103,54 s, 48,6 MB/s

real    18m25,257s
user    0m0,008s
sys     0m0,000s
Executing: time umount /mnt

real    6m24,581s
user    0m0,003s
sys     0m0,003s
13

beside all the software-related issues. i’m sure that the external device isn’t designed for this and the thermal stress adds additional file-transfer errors that the system has to correct. this correction-procedures (rewriting/syncing) drops the transfer-rate too.

1 Like
14

The result of the triage above show a fairly consistent result even when using different devices and filesystems. The only one that really stands out as really bad … 18m25s to copy 50G and then 6m24s to unmount - that is the Kingston DataTraveler 64G

Perhaps the issue is with the combination of rsync and ntfs?

rsync -rvh --size-only --progress /data/ntfs1 /data/ntfs2
15

Thank you. You tests are for SSD drives, using USB-C whereas mine are merely 3.0 devices using USB type A.

Here is the output of test.sh after changing the udev rules.

bash test.sh
Executing: time dd if=/dev/urandom of=/mnt/test.img bs=1G count=100 status=progress
107374182400 bytes (107 GB, 100 GiB) copied, 2604 s, 41.2 MB/s
100+0 records in
100+0 records out
107374182400 bytes (107 GB, 100 GiB) copied, 2604.24 s, 41.2 MB/s

real 43m24.495s
user 0m0.008s
sys 7m33.937s
Executing: time umount /mnt

real 0m1.465s
user 0m0.006s
sys 0m0.690s

iotop reported about 42MB/s, i guess for the overhead.
This is good news, it means linux can in fact write to the flash drive at 41.2MB/s and that 5MB/s that everyone puts up with on every other forum is in fact NOT NORMAL.

43 min for 100GB. These are proper speeds …well in the case of 2.0.

That got me thinking… This flash drive is plugged into a 3.0 port, but these are 2.0 speeds. I checked lsusb and sure enough: Bus 3 Dev 2 is not on a 3.0 port. But it is plugged in to a 3.0 port just like the other LiveUSB drive (same exact flash drive from store) which is Bus 4 Dev 5, listed as a 3.0 device noting the 5000M, which means it is not the flash drive but the port being improperly configured. I double checked, it is a blue 3.0 port with the extra pins beside the other one.

>~ lsusb  
Bus 002 Device 002: ID 8087:8000 Intel Corp. Integrated Rate Matching Hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 002: ID 8087:8008 Intel Corp. Integrated Rate Matching Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 004 Device 005: ID 23a5:3379 USB Disk 3.0
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 003 Device 003: ID c0f4:01f5 Usb KeyBoard Usb KeyBoard
Bus 003 Device 005: ID 248a:8363 Maxxter Wireless Receiver
Bus 003 Device 002: ID 23a5:3379 USB Disk 3.0
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

> ~ lsusb -t     
/:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 5000M
    |__ Port 2: Dev 5, If 0, Class=Mass Storage, Driver=usb-storage, 5000M
/:  Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/10p, 480M
    |__ Port 1: Dev 2, If 0, Class=Mass Storage, Driver=usb-storage, 480M
    |__ Port 3: Dev 5, If 0, Class=Human Interface Device, Driver=usbhid, 12M
    |__ Port 3: Dev 5, If 1, Class=Human Interface Device, Driver=usbhid, 12M
    |__ Port 4: Dev 3, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
    |__ Port 4: Dev 3, If 1, Class=Human Interface Device, Driver=usbhid, 1.5M
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/2p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/6p, 480M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/2p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M

I did a real world test using Dolphin and monitored with iotop. I transferred 105GB from the internal SATA HDD to the flash drive. While it does now incorporate short bursts up to 30MB/s, it mostly remains at 300KB-5MB/s. The occasional bursts do increase the amount transferred, but this is barely a bandaid solution at best. I stopped the test after 1 hour (judging by the clock 2:25 to 3:25AM) and it has moved 30.3GB of 105GB which averages out to about 8MB/s. Still a lousy number, about 25% of the actual capable speed when accounting for overhead.

However, The initial dump of files transferred at almost quickly up to around 17GB.
The first 17GB was transferred within the first 20 minutes
Between 17 and 22 it started to slow down
The remaining 13GB took 40min.
Perhaps this will give clues?
This would average out to about 14.5MB/s for the first 20 minutes, then ~5MB/s for the remaining 40 minutes which once again, is not normal speeds.

I am going to reboot, and boot the LiveISO on the 2.0 port on the front of the machine and use the 3.0 port that the LiveISO occupied with the USB 3.0 flash drive instead, re add the udev rules list, and rerun these tests.

I can retry rsync and the external 3.0 HDD as well using this method once I move the LiveISO drive.

Hopefully this is enough proof so far to show that there is indeed a problem, but i will retry with it on the 3.0 slot to make sure.

Thanks everyone!

16

I never said it wasn’t a problem …

The main problem is users errornously expecting Linux === Windows.

USB ports on the same controller shares power and bandwidth, which - in case of two devices - roughly would result in a halved transfer rate.

That is not a kernel issue …

And furthermore - specifications rarely matches realworld …

17

beside all the software-related issues. i’m sure that the external device isn’t designed for this and the thermal stress adds additional file-transfer errors that the system has to correct. this correction-procedures (rewriting/syncing) drops the transfer-rate too.

Again, not trying to be rude, but this is an excuse against real world tests.

It works fine under windows at the correct speed and transfers all the data and can even be safely removed no problem, and so far has worked when running dd with the udev rules given by linux-aarhus, but not in a real test on linux transferring actual files which leads me to believe it’s a bug.

The only one that really stands out as really bad … 18m25s to copy 50G and then 6m24s to unmount - that is the Kingston DataTraveler 64G

My point exactly. I pulled that drive up. It is a USB 3.x (Not type C) Flash drive. On average, your drive is operating at 50GB(50000) divided by 25min (18.5+6.5) for about 33MB/s. It’s a rough estimate, but you see what I am getting at.

Those are not 3.0 speeds, it should be hitting above 80-150MB/s. I’m betting when I restart and use my flash drive in the 3.0 ports properly it will write at that speed like yours, about 33MB/s, which is still incredibly low, but better than 5MB/s.

Thanks for the extra data!

18

The main problem is users errornously expecting Linux === Windows.

True, but this has been an issue for almost two decades now

USB ports on the same controller shares power and data, which - in case of two devices - roughly would result in a halved transfer rate.

I will redo the test with the drives in different positions leaving the 3.0 port open for the 3.0 flash drive only

That is not a kernel issue …

it might be :grinning:

And furthermore - specifications rarely matches realworld …

That’s understandable.
What is not is less than 20% actual throughput in real world tests when in staged tests using dd it works as intended.

Sorry we are posting in between each other

19

How…How the **** can the software (the kernel is software after all) have any effect on the physical hardware’s capabilities?

And it will always be the way it is. An “issue” as you call it. Or rather, it will always be an issue until there is a way for software to influence hardware capabilities…

1 Like
20

I didn’t read much here and skip some posts.

Do you use Swap?

Check inxi
inxi --admin --usb --cpu --machine --disk --partitions --swap

21

What you see could be the effect of the kernel cache.

If you want to draw any conclusions on a sudden drop it could be the stick’s internal software doing error correction, which then point to possible defective memory cells.

What you should look at is the final numbers.

The amount of time taken to write the data and the amount of time taken to unmount the device - which would include flush what remains of the buffer.

What happens inbetween is skewed by cache is not a reliable number as it will be different running the same test twice.

22

Please, try to focus on solving your specific issue and avoid negative general assessments which could be deemed as rant against Linux.

1 Like
23

I tried to rsync my data from my internal Nvme with Ext4 to my external USB stick 64 GB with Exfat via USB 3.0.

I created a shell script.

You need to replace <SOURCE> with your file path, <TARGET> too.

#!/bin/bash

echo "===Start==="
echo "RAM cache:"
vmstat

for i in {1..8}; do
    printf "\n===%s.repeat===\n" "$i"
    echo "Rsync your data:"
    rsync --progress "<SOURCE>" "<TARGET>/$i"
done

echo ""
echo "===End==="
echo "RAM cache:"
vmstat

Result:

===Start===
RAM cache:
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0      0 24244052  61712 2351780    0    0 17946  1531 10831    7  2  2 95  1  0

===1.repeat===
Rsync your data:
video_test_2GB.mkv
  2.108.896.285 100%  518,21MB/s    0:00:03 (xfr#1, to-chk=0/1)

===2.repeat===
Rsync your data:
video_test_2GB.mkv
  2.108.896.285 100%  675,34MB/s    0:00:02 (xfr#1, to-chk=0/1)

===3.repeat===
Rsync your data:
video_test_2GB.mkv
  2.108.896.285 100%  138,89MB/s    0:00:14 (xfr#1, to-chk=0/1)

===4.repeat===
Rsync your data:
video_test_2GB.mkv
  2.108.896.285 100%   55,68MB/s    0:00:36 (xfr#1, to-chk=0/1)

===5.repeat===
Rsync your data:
video_test_2GB.mkv
  2.108.896.285 100%   57,13MB/s    0:00:35 (xfr#1, to-chk=0/1)

===6.repeat===
Rsync your data:
video_test_2GB.mkv
  2.108.896.285 100%   24,94MB/s    0:01:20 (xfr#1, to-chk=0/1)

===7.repeat===
Rsync your data:
video_test_2GB.mkv
  2.108.896.285 100%   24,53MB/s    0:01:22 (xfr#1, to-chk=0/1)

===8.repeat===
Rsync your data:
video_test_2GB.mkv
  2.108.896.285 100%   26,23MB/s    0:01:16 (xfr#1, to-chk=0/1)

===End===
RAM cache:
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 2  1    256 430224  36460 25875708    0    0  7662 34079 4241    3  1  1 88 10  0

RAM showed small free space after the transfer.
I know now that the transfer speed depends on the temperature of my USB stick.
When temperature is higher, speed will be reduced more dramatically.

I would suggest a solution. The USB stick needs a pause of 1 min due to overheating after transferring data 10 GB. After the pause then it goes faster.

Edit:// The cache transfer was used in my test, which is a trick. My test is unfair.

1 Like
24

i already pointed out that a lot of usb-storages really suffer due to thermal problems, the resulting transfer-errors and the additional error-correction that must be done with more syncing, but the TO declined. there is one pro-argument of the TO that he reports no issues with MS-Windows.
there is in fact a different between the handling of large file-transfers between windows and linux.

25

this is a interesting thread. i was googeling for a long time. can it be that one main problem is the synced and cached transfer to the usb? i’ve got no usb-mass-media for testing this but if anyone can test to mount the usb-device in async mode and flush to a folder, copy a large amount of data inside this and check if the speed doesn’t collapse ?

mount -o async,flush -t ntfs /dev/sdb1 ~/somewhere

(fuse-exfat package is needed)

26

You are right after my tests. The cached transfer is a trick.

sudo mount -t exfat -o async,uid=1000,gid=1000 /dev/sda1 /mnt/

With cache:
Result:

===Start===
RAM cache:
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 2  0      0 18555080  86648 6537648    0    0  3640  6544 4136    3  1  1 97  2  0

===1.repeat===
Rsync your data:
video_test_2GB.mkv
  2.108.896.285 100%  674,21MB/s    0:00:02 (xfr#1, to-chk=0/1)

===End===
RAM cache:
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0      0 16010948  86648 9024236    0    0  3632  6530 4149    3  1  1 97  2  0

I disabled cache:

echo 16777216 > /sys/block/sda/bdi/max_bytes
echo 1 > /sys/block/sda/bdi/strict_limit
Result:

===Start===
RAM cache:
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0      0 18078912  94088 6702120    0    0  2958  6543 4306    3  1  1 97  1  0

===1.repeat===
Rsync your data:
video_test_2GB.mkv
  2.108.896.285 100%   37,40MB/s    0:00:53 (xfr#1, to-chk=0/1)

===End===
RAM cache:
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0      0 15544208  94204 9188888    0    0  2870  7507 4267    3  1  1 97  2  0
1 Like