Slow USB transfers

zippydippy · 25 August 2023 07:34

Hi, I’m new here and need some help! I am reporting this as what I think is a kernel bug, and perhaps this will go through the proper channels. I have had this problem since probably Fedora 7, back in the mid 2000’s. I even recently experienced it on Kubuntu. I now run Windows 7 (10 was the nail in the coffin,) Manjaro, and EndeavourOS on different machines.

This singular problem is preventing me from running linux full time, or recommending it to friends/family.

I am trying to copy 260+ GB of data from a SATA hard drive to either

An external SATA 2TB hard drive via USB 3.0
An external 128 GB USB 3.0 flash drive (I know this is too small, I only attempted 116GB of data here)
(Edited usb flash drive capacity after rereading what i wrote. 138gb? lol)
In both cases, transfers slow to a crawl. I’m talking 4MB down to 250KB/s. Not good when trying to move 260 gigs.

The drive they are transferring from is a SATA drive
I am using 3.0 ports
They are both 3.0 devices
I have tried increasing dirty bytes
I have tried copying via Doplhin
I have tried cp command
I have tried rsync
I am running on a Live Manjaro KDE ISO on one of the 3.0 ports
(I also tried it on an a machine with manjaro installed, same effect.)

The same thing always happens, it transfers anywhere from 10-70 GB at ~100MB/s, then progressively slows down until hitting 4MB-250KB/s.

I am not talking about the burst of speed incorrectly reported while filling up RAM or cache, which would be irrelevant due to the 70GB that transferred quickly. I am running a Live ISO on 8 GB of ram.

The reason why I am trying to report this as a kernel bug is because it has always existed (search the web for it,) and unless you switch back to windows you may not notice it.

I know this is a bug because: the first 70 GB of data transferred is complete and intact in a VERY short time (definitely less than 30 minutes.) All files open, All files play, etc etc. I stopped the transfer when it became slow and made sure I could safely detach the drive, meaning the files were transferred. On average I would say it was about 75 MB/s transfer speed.

I restarted rsync, and the remaining 12 hours I waited also copied files, but in the end was about 105 GB of 260 GB total. It bounces between 4 MB and 250KB/s. If you search “Slow USB speed” with “linux,” “ubuntu,” etc, you can find literally hundreds of posts highlighting this problem. The question always goes unanswered. (Except expanding dirty bytes, but that seems to only delay the inevitable.)

It makes no difference whether it is the flash drive or hard drive, both start at the same speed, and slow down to the same speed. If I switch to windows, it works as expected. I can copy all of it in about an hour. But I am trying to do this without windows, so I deleted the data and am retrying over and over.

So to reiterate, windows can transfer all of it in about an hour. So please dont say
“slow speeds is normal for USB drives”
“your USB drive is old”
“your drive is a knockoff” (I have bought one before though lol)
“its the cache filling up”
“5mb/s is good for 3.0”
“blame microsoft”
as those are the usual responses.

This is in no way a stab at Manjaro, as this has existed for almost 20 years at this point.
I really appreciate the work you all do here!
But if anyone can help me it would be greatly, greatly appreciated! I would like to get to the bottom of this 20 year old bug. Thanks in advance!

Olli · 25 August 2023 07:48

here you are:

zippydippy · 25 August 2023 07:56

Thanks, but this does not fix it. I already mentioned altering dirty bytes. It does help with the speed initially, and does help with proper reporting of transfer speeds, but does nothing in regard to fixing the progressive slowness. This has occurred for 20 years now, and the transfer works fine in windows.

Xephon · 25 August 2023 09:20

Yes, there is a problem with USB transfer speeds in linux, and as far as I understand nobody has a slightest idea on how to fix it.

For SSDs you could try to use a SATA mobile rack. Or even consider permanently installing a small SSD inside your PC case for regular backups.
But there is no solution for USB sticks at the moment.

megavolt · 25 August 2023 09:49

Hello @zippydippy

Since you tested a lot, the only thing left to mention would be “SMR against CMR” on HDDs. SMR uses so called zones, which is supported by the filesystems like BTRFS to mitigate these “speed drops”. But even if supported, it is known to drop transfer speed with big files.

Since you are new, but not a novice (so it seems): Could you share real test data and information about the hardware?

Maybe someone can reproduce your issue and narrow it down further.

I use also USB3 and HDDs/FlashDrives, but I never experience your issue. Yeah after an amount of data it it drops sometimes to ~50MB/s and goes higher and lower, but I use BTRFS with DUP Data profile, therefore the data is written twice, while it uses the full bandwidth it displays about the half.

It start with ~100MB/s
At about 8-10GB it stucks and drops to about 60-70MB/s
In iotop (i use iotop-c from AUR), I see about 100-200MB/s current disk write speed and ~100MB/s total speed.
dd still reports 60-70MB/s

I used:

dd if=/dev/urandom of=/media/WDElements/testfile bs=512 conv=noerror status=progress

and a second terminal:

 iotop -Pox

Take into account that it writes twice on the same HDD (data dup profile) and uses zstd=9 compression. But at the end I get zero speed drop. It stays like this…

As you see ~100Gb in ~25min (1542sec) I believe it was even faster before without compression and dup profile.

WDElements

sudo lsusb -v -d  1058:25a3

Bus 002 Device 002: ID 1058:25a3 Western Digital Technologies, Inc. Elements Desktop (WDBWLG)
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               3.10
  bDeviceClass            0 
  bDeviceSubClass         0 
  bDeviceProtocol         0 
  bMaxPacketSize0         9
  idVendor           0x1058 Western Digital Technologies, Inc.
  idProduct          0x25a3 Elements Desktop (WDBWLG)
  bcdDevice           10.31
  iManufacturer           1 Western Digital
  iProduct                2 Elements 25A3
  iSerial                 3 334A475652363147
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength       0x002c
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          0 
    bmAttributes         0xc0
      Self Powered
    MaxPower                8mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           2
      bInterfaceClass         8 Mass Storage
      bInterfaceSubClass      6 SCSI
      bInterfaceProtocol     80 Bulk-Only
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0400  1x 1024 bytes
        bInterval               0
        bMaxBurst              15
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x02  EP 2 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0400  1x 1024 bytes
        bInterval               0
        bMaxBurst              15
Binary Object Store Descriptor:
  bLength                 5
  bDescriptorType        15
  wTotalLength       0x0016
  bNumDeviceCaps          2
  USB 2.0 Extension Device Capability:
    bLength                 7
    bDescriptorType        16
    bDevCapabilityType      2
    bmAttributes   0x00000f0e
      BESL Link Power Management (LPM) Supported
    BESL value     3840 us 
  SuperSpeed USB Device Capability:
    bLength                10
    bDescriptorType        16
    bDevCapabilityType      3
    bmAttributes         0x00
    wSpeedsSupported   0x000e
      Device can operate at Full Speed (12Mbps)
      Device can operate at High Speed (480Mbps)
      Device can operate at SuperSpeed (5Gbps)
    bFunctionalitySupport   1
      Lowest fully-functional device speed is Full Speed (12Mbps)
    bU1DevExitLat          10 micro seconds
    bU2DevExitLat          32 micro seconds
Device Status:     0x000d
  Self Powered
  U1 Enabled
  U2 Enabled

So you mean then NTFS? It isn’t a surprise to me since NTFS works on user space (ntfs-3g). It is in general slower, same as ext4 driver on Windows.

That is an indicator for an SMR HDD, but can be wrong. CMR HDDs, don’t have that issue.

Also cheap fake USB Flash-Drives, which reports more storage, then actually there can be the issue here. It is known that those devices starts fast and when they reach a limit then they transfer in slow speed or even nothing or produce corrupted files after that.

So please check that first to out rule these common problems.

linux-aarhus · 25 August 2023 12:58

Please use the search function.

Search results for 'usb transfer speed' - Manjaro Linux Forum

It has been dicussed on numerous occasions - one of which is only a few days old.

Linux kernel caches filesystem to improve performance. What you are experiencing is the fact that is take longer to write to your USB than it takes to read from the source.

This build up a buffer which needs to be emptied before the stick can be removed.

How fast the buffer is written to disk depend on your system, configuration and the usb stick.

You can use one of the numerous suggestions - what works best is hard to say.

[root tip] [How To] Disable write cache for USB storage devices

zippydippy · 25 August 2023 16:23

(Sorry I can’t reply to more than 2 at once because I’m new, so I’m cutting you out Xephon! )
At Xephon: Sadly, you may be correct.

@linux-aarhus Not to be rude, but please reread my post. It is not the cache/buffer. That’s not what I am talking about. I am aware that when dolphin says my transfer speeds are above 100 MB/s that it is merely the buffer filling up. But the emptying of that buffer should be on par with the write speed of the device, which it is not. Compare it to windows, which sustains ~80-100MB/s continuously and is done in a little over an hour.

I double checked your search link, the post that is a few days old here

(I removed link because I cant post it, I’m too new here.)

actually reinforces my point. This is the same issue as mine. 38+ hours for a 274GB transfer at ~5MB/s is insane for USB 3.0, or even 2.0. I mentioned in the OP that on windows it takes maybe an hour and 15 minutes. That is a far cry from 1 and a half days. This must be a bug of some sort. It is also mentioned on Ubuntu and Fedora forums dating back to mid 2000’s which is when I first experienced it.

I read your response here and on that page as well, pointing to modifying the udev rules. Is modifying the udev rules to disable the write-cache the same as shrinking or increasing dirty bytes? I will try it if it might help and will post the results. Is there a way to do it without rebooting? I am on a Live ISO.

@megavolt: I installed iotop and ran sudo iotop -Po (-x errored out as an unavailable command) I did not use dd yet since it is still transferring. I figured it may as well be the same anyway. But I will run dd if you need a comparison of some sort.

Here is the current result and speeds. rsync has been running since about 2 AM here, so 9 hours. Rsync confirms this. It has only transferred 43 GB. When i first got home, it was bouncing between 250-700 KB/s. It’s running around 1.3MB/s at the moment.
(sigh…screenshot removed because I cant post media yet)
here is the copy and pasted outputs of 2 terminals…

iotop:

Total DISK READ :    1235.78 K/s | Total DISK WRITE :     813.52 K/s
Actual DISK READ:    1235.78 K/s | Actual DISK WRITE:       0.00 B/s
    PID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND                            
   2310 be/4 root     1235.78 K/s    0.00 B/s  ?unavailable?  mount.ntfs /dev/sd~mes,uhelper=udisks2
   5000 be/4 root        0.00 B/s  813.52 K/s  ?unavailable?  mount.ntfs /dev/sd~mes,uhelper=udisks2

rsync:

Users/USERNAME/Desktop/New folder/2017-12-01 iphone photos up until 11-30-17/iphone photos up until 11-30-17 12224.JPG
 42,766,260,000  24%    1.18MB/s    9:35:39 (xfr#43730, ir-chk=12272/165340)
Users/USERNAME/Desktop/New folder/2017-12-01 iphone photos up until 11-30-17/iphone photos up until 11-30-17 12225.MOV
 42,795,718,432  24%  708.32kB/s   52:31:19

I do not know how to use quotes here yet.
At megavolt said:

So you mean then NTFS? It isn’t a surprise to me since NTFS works on user space (ntfs-3g). It is in general slower, same as ext4 driver on Windows.

The drive im transferring from is NTFS, yes. It is an internal SATA drive.
I am transferring to either NTFS external drive over USB 3.0 (internal drive itself is SATA)
OR to a USB 3.0 stick that is exFAT.

I would blame ntfs-3g, but that doesn’t explain the perfect speeds and huge data transfer during the first 30 minutes (up to 70GB.) After which it slows to a crawl.

That is an indicator for an SMR HDD, but can be wrong. CMR HDDs, don’t have that issue.

Again, It also happens with the USB flash drive. Anywhere from 10-70GB at ~80-100MB/s speeds, then a drop to <5MB/s. I believe this plus the fact that windows has no problem at all in either case would rule out SMR.

EDIT: ~80-100MB/s were for the external HDD, not the USB flash drive which is closer to probably 45MB/s on windows

Also cheap fake USB Flash-Drives, which reports more storage, then actually there can be the issue here. It is known that those devices starts fast and when they reach a limit then they transfer in slow speed or even nothing or produce corrupted files after that.

It was a cheap drive (Got it on sale ) and I have bought fake ones before. I had a 128 GB drive that was actually a 16GB flash drive that would “wrap around” and overwrite the beginning, causing some lovely corruption for family photos. This current drive is legit.

Thanks again for the replies everyone. As I’ve said, both the flash and external drive work fine under windows at the proper speeds without any corruption.

Edit: lol my quotes worked

linux-aarhus · 25 August 2023 16:35

I don’t consider you rude - if you are 100% certain it is not cache related, and not hardware related - then I have no idea.

The thing is - I cannot reproduce such issue - and a lot of things can be done with configuration - but the thing is - there is no one-size-fits-all solution … what works for me might not work for you - however the options presented in one of the linked topics has proved to be working for some.

megavolt · 25 August 2023 17:29

If you believe that, then I have no idea here. I don’t see such an issue with my drives and I cannot reproduce it. It works as intended.

Maybe check if the kernel has detected a zoned HDD:

 cat /sys/block/sd[a-z]/queue/zoned

Maybe here you get better information about SMR: Getting Started with SMR Hard Disks | Zoned Storage

At this stage only f2fs or btrfs have optimization for SMR on linux. No NTFS or exFAT.

I can imagine that a USB3 flash drive could get hot and heat can also throttle speed.

If possible try linux filesystem: ext4, btrfs, xfs etc.

However… I strongly believe that this is not an usb driver issue at all on linux. Maybe your mainboard has a special USB3 mode, who knows? Like USB 3.0 Boost or Turbo Mode or UASP Mode ? Maybe that comes into play here? Just puzzling.

NTFS and EXFAT are both user space by default.

Wollie · 25 August 2023 18:05

I can confirm what @megavolt and @linux-aarhus stated, I have never seen any nameable performance differences between Windoze 7, 10 and Linux (Archlinux, Manjaro, EndeavourOS) with any type of USB stick or any external HDD, either using USB 2.0, 3.1 or 3.2 as long as NTFS is not the target filesystem with the known limitations of ntfs-3g.

I’m pretty sure your issues have something to do with your specific environment and settings. So, @zippydippy, @Xephon, please stop claiming there would be a general issue with USB performance in Linux if you cannot prove it.

zippydippy · 26 August 2023 05:05

So I tried doing the same thing on EndeavourOS, different machine. Ryzen 1800x, 16GB ram, 2tb Mechanical SATA HDD, USB 3.0 ports (everything else removed so no chance of 2.0 speeds) and the same thing happens using a completely different ext4 source drive with large files. Fast, followed by a drop to ~1MB/s.

I tested it again on a laptop of mine with 3.0 ports. Also formatted with ext4, and transferring to the NTFS drive or USB stick. Same results: ~1MB/s

@megavolt

Maybe check if the kernel has detected a zoned HDD:
Output:
cat /sys/block/sd[a-z]/queue/zoned 
none
none
none
none
none

Changing filesystems will not help in my case, I need to keep the external HDD as NTFS. The USB stick is exFAT, so no SMR, but I may reformat it to ext4 to see if it makes a difference after I modify udev next. Since both drives drop speeds, I’m not completely ruling out it being a filesystem issue.

I strongly believe that this is not an usb driver issue at all on linux.

I believe it is either a usb driver bug or kernel bug. I have consistently retested this across 3 machines now, including EndeavourOS.

@Wollie

Hi

I can confirm what megavolt and linux-aarhus stated, I have never seen any nameable performance differences between Windoze 7, 10 and Linux (Archlinux, Manjaro, EndeavourOS) with any type of USB stick or any external HDD, either using USB 2.0, 3.1 or 3.2 as long as NTFS is not the target filesystem with the known limitations of ntfs-3g.

Please, anyone here, try this. Try it on your current OS, or a Live one, copy a Giant Folder (above 100GB) from an internal drive to an external 3.0 drive and post your results. Use Dolphin, cp, rsync, whatever works for you. Because the more I read this the less I am actually believing it. It is fast, but it will drop off to less than 10% theoretical speeds after some unknown amount which means it is indeed a bug of some sort.

I’m pretty sure your issues have something to do with your specific environment and settings. So, zippydippy, Xephon, please stop claiming there would be a general issue with USB performance in Linux if you cannot prove it.

This is the reason a lot of would be linux users are driven away. I have an issue and it is dismissed as if the situation is not real and reproducible. Granted, it may not be reproducible on certain hardware. However, I have proved it across 3 machines and two arch Linux distros, with experience of it happening in the past using both Kubuntu and Fedora using 2 other machines (modern at the time) with many posts online reiterating this experience. None of these hardware are similar and the proof has definitely been provided.

I keep reading online that there is no proof, when there clearly is in the case of slow USB transfers and never in the “it works for me” case. My claim stands, it is a real phenomenon probably coming from the kernel or USB drivers.

Please try it for yourself and post results. Copy a giant folder and post the speeds and the experience all the way to the finish line. Using an external flash/HDD is very commonplace and this is common enough situation for a bug report to be warranted.

Btw, I am still transferring that folder using rsync on the first pc. it is at 167GB out of 262GB and has taken over 22 hours with a current rate of 300KB/s. This is the same folder that took about an hour and 15 min on windows.

linux-aarhus I will attempt the udev fix next to completely rule out the cache issue. Then I will try reformatting the USB to ext4

Sorry for the long post, please stick with me.

edited for typos

linux-aarhus · 26 August 2023 05:32

Even this is no scientific test - the results speaks for themselves.

You can draw your own conclusions - to me it is evident the LInux kernel is not the culprit …

 $ inxi -SC
System:
  Host: tiger Kernel: 6.5.0-rc7-next-20230824-1-next-git-12477-g2b3bd393093b
    arch: x86_64 bits: 64 Desktop: KDE Plasma v: 5.27.7 Distro: Manjaro Linux
CPU:
  Info: 12-core model: AMD Ryzen Threadripper PRO 5945WX s bits: 64
    type: MT MCP cache: L2: 6 MiB
  Speed (MHz): avg: 2269 min/max: 1800/7015 cores: 1: 1800 2: 1800 3: 1800
    4: 1800 5: 1800 6: 2940 7: 1739 8: 1800 9: 1800 10: 4100 11: 1800 12: 2940
    13: 1800 14: 1740 15: 1800 16: 1800 17: 1800 18: 4100 19: 1800 20: 1756
    21: 1757 22: 4100 23: 1800 24: 4100

Depending on your system and on the source - e.g. another disk device - reproducing will be different as these tests does not read another disk device but a stream of random bytes produced by the system. The reason for using thr random stream is simply to avoid identical streams when testing the various devices.

Reproduce the tests

Ensure your system is fully up-to-date and hdparm installed

sudo pacman -Syu hdparm

Create this udev rule (referenced earlier)

 $ cat /etc/udev/rules.d/99-usb-sync.rules 
# rule to disable write cache for usb storage
# requires hdparm to be installed
ACTION=="add|change", KERNEL=="sd[a-z]", ENV{ID_USB_TYPE}=="disk", RUN+="/usr/bin/hdparm -W 0 /dev/%k"

#
# the following rules is introduced with kernel 6.2
# https://docs.kernel.org/admin-guide/abi-testing.html#abi-sys-class-bdi-bdi-strict-limit
# https://docs.kernel.org/admin-guide/abi-testing.html#abi-sys-class-bdi-bdi-max-ratio
# https://docs.kernel.org/admin-guide/abi-testing.html#abi-sys-class-bdi-bdi-max-bytes
ACTION=="add|change", KERNEL=="sd[a-z]", ENV{ID_USB_TYPE}=="disk", RUN+="/usr/bin/echo 1 > /sys/block/%k/bdi/strict_limit",  RUN+="/usr/bin/echo 50 > /sys/block/%k/bdi/max_ratio", RUN+="/usr/bin/echo 16777216 > /sys/block/%k/bdi/max_bytes"

Reload the rules

sudo udevadm control --reload

Create a test script - requires the package time to be installed.

 $ cat test.sh
#!/usr/bin/env bash
echo "Executing: time dd if=/dev/urandom of=/mnt/test.img bs=1G count=100 status=progress"
time sudo dd if=/dev/urandom of=/mnt/test.img bs=1G count=100 status=progress
echo "Executing: time umount /mnt"
time sudo umount /mnt

You can create variations over this to simulate other conditions e.g.

# create 100 files of 100M each
for i in {1..100}; do
    sudo dd if=/dev/urandom of="/mnt/test${i}.img" bs=10M count=10
done

Mount the test device/partition on /mnt and execute …

sudo mount /dev/sdxY /mnt

SATA enclosure test

OCZ Agility 3 240G SATA in 1.gen USB 3 enclosure (? 2010)

formatted with ext4

 $ sudo bash test.sh
Executing: time dd if=/dev/urandom of=/mnt/test.img bs=1G count=100 status=progress
107374182400 bytes (107 GB, 100 GiB) copied, 664 s, 162 MB/s
100+0 records in
100+0 records out
107374182400 bytes (107 GB, 100 GiB) copied, 735,558 s, 146 MB/s

real    12m15,586s
user    0m0,005s
sys     0m0,006s
Executing: time umount /mnt

real    0m9,505s
user    0m0,004s
sys     0m0,008s

formatted with ntfs

 $ sudo bash test.sh
Executing: time dd if=/dev/urandom of=/mnt/test.img bs=1G count=100 status=progress
107374182400 bytes (107 GB, 100 GiB) copied, 676 s, 159 MB/s
100+0 records in
100+0 records out
107374182400 bytes (107 GB, 100 GiB) copied, 675,717 s, 159 MB/s

real    11m15,742s
user    0m0,011s
sys     0m0,000s
Executing: time umount /mnt

real    0m59,460s
user    0m0,004s
sys     0m0,007s

USB-C device test

Samsung 1.8T T7 Shield USB-C (july 2023)

formatted with ntfs

 $ sudo bash test.sh
Executing: time dd if=/dev/urandom of=/mnt/test.img bs=1G count=100 status=progress
107374182400 bytes (107 GB, 100 GiB) copied, 485 s, 221 MB/s
100+0 records in
100+0 records out
107374182400 bytes (107 GB, 100 GiB) copied, 485,176 s, 221 MB/s

real    8m5,205s
user    0m0,004s
sys     0m0,007s
Executing: time umount /mnt

real    0m15,391s
user    0m0,002s
sys     0m0,007s

This device proved a 3-4 minute improvement in transfer speed compared to the old SATA device

WD Passport 512G MILAN II USB-C (? 2021)

formatted with ext4

 $ subo bash test.sh
Executing: time dd if=/dev/urandom of=/mnt/test.img bs=1G count=100 status=progress
107374182400 bytes (107 GB, 100 GiB) copied, 619 s, 174 MB/s
100+0 records in
100+0 records out
107374182400 bytes (107 GB, 100 GiB) copied, 685,141 s, 157 MB/s

real    11m25,165s
user    0m0,012s
sys     0m0,000s
Executing: time umount /mnt

real    0m6,726s
user    0m0,010s
sys     0m0,000s

64G USB stick test

SanDisk Extreme 64G USB 3.0 (? 2020)

This is tested like the others but using a 50G stream

 $ cat test-50.sh
#!/usr/bin/env bash
echo "Executing: time dd if=/dev/urandom of=/mnt/test.img bs=1G count=100 status=progress"
time sudo dd if=/dev/urandom of=/mnt/test.img bs=1G count=100 status=progress
echo "Executing: time umount /mnt"
time sudo umount /mnt

formatted with exfat

 $ sudo bash test-50.sh
Executing: time dd if=/dev/urandom of=/mnt/test.img bs=1G count=50 status=progress
53687091200 bytes (54 GB, 50 GiB) copied, 869 s, 61,8 MB/s
50+0 records in
50+0 records out
53687091200 bytes (54 GB, 50 GiB) copied, 868,694 s, 61,8 MB/s

real    14m28,720s
user    0m0,004s
sys     0m0,004s
Executing: time umount /mnt

real    2m19,978s
user    0m0,001s
sys     0m0,005s

formatted with ntfs

 $ sudo bash test-50.sh
Executing: time dd if=/dev/urandom of=/mnt/test.img bs=1G count=50 status=progress
53687091200 bytes (54 GB, 50 GiB) copied, 740 s, 72,5 MB/s
50+0 records in
50+0 records out
53687091200 bytes (54 GB, 50 GiB) copied, 740,327 s, 72,5 MB/s

real    12m20,350s
user    0m0,002s
sys     0m0,006s
Executing: time umount /mnt

real    3m11,227s
user    0m0,006s
sys     0m0,000s

formatted using f2fs

 $ sudo bash test-50.sh
Executing: time dd if=/dev/urandom of=/mnt/test.img bs=1G count=50 status=progress
53687091200 bytes (54 GB, 50 GiB) copied, 701 s, 76,6 MB/s
50+0 records in
50+0 records out
53687091200 bytes (54 GB, 50 GiB) copied, 700,81 s, 76,6 MB/s

real    11m40,822s
user    0m0,001s
sys     0m0,008s
Executing: time umount /mnt

real    2m35,335s
user    0m0,000s
sys     0m0,006s

Kingston DataTraveler100 64G USB 3.1 (? 2021)

This device had been tested to showcase vendor quality

formatted using ntfs

 $ sudo bash test-50.sh
Executing: time dd if=/dev/urandom of=/mnt/test.img bs=1G count=50 status=progress
53687091200 bytes (54 GB, 50 GiB) copied, 1104 s, 48,6 MB/s
50+0 records in
50+0 records out
53687091200 bytes (54 GB, 50 GiB) copied, 1103,54 s, 48,6 MB/s

real    18m25,257s
user    0m0,008s
sys     0m0,000s
Executing: time umount /mnt

real    6m24,581s
user    0m0,003s
sys     0m0,003s

Olli · 26 August 2023 07:55

beside all the software-related issues. i’m sure that the external device isn’t designed for this and the thermal stress adds additional file-transfer errors that the system has to correct. this correction-procedures (rewriting/syncing) drops the transfer-rate too.

linux-aarhus · 26 August 2023 08:12

The result of the triage above show a fairly consistent result even when using different devices and filesystems. The only one that really stands out as really bad … 18m25s to copy 50G and then 6m24s to unmount - that is the Kingston DataTraveler 64G

Perhaps the issue is with the combination of rsync and ntfs?

rsync -rvh --size-only --progress /data/ntfs1 /data/ntfs2

zippydippy · 26 August 2023 08:41

Thank you. You tests are for SSD drives, using USB-C whereas mine are merely 3.0 devices using USB type A.

Here is the output of test.sh after changing the udev rules.

bash test.sh
Executing: time dd if=/dev/urandom of=/mnt/test.img bs=1G count=100 status=progress
107374182400 bytes (107 GB, 100 GiB) copied, 2604 s, 41.2 MB/s
100+0 records in
100+0 records out
107374182400 bytes (107 GB, 100 GiB) copied, 2604.24 s, 41.2 MB/s

real 43m24.495s
user 0m0.008s
sys 7m33.937s
Executing: time umount /mnt

real 0m1.465s
user 0m0.006s
sys 0m0.690s

iotop reported about 42MB/s, i guess for the overhead.
This is good news, it means linux can in fact write to the flash drive at 41.2MB/s and that 5MB/s that everyone puts up with on every other forum is in fact NOT NORMAL.

43 min for 100GB. These are proper speeds …well in the case of 2.0.

That got me thinking… This flash drive is plugged into a 3.0 port, but these are 2.0 speeds. I checked lsusb and sure enough: Bus 3 Dev 2 is not on a 3.0 port. But it is plugged in to a 3.0 port just like the other LiveUSB drive (same exact flash drive from store) which is Bus 4 Dev 5, listed as a 3.0 device noting the 5000M, which means it is not the flash drive but the port being improperly configured. I double checked, it is a blue 3.0 port with the extra pins beside the other one.

>~ lsusb  
Bus 002 Device 002: ID 8087:8000 Intel Corp. Integrated Rate Matching Hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 002: ID 8087:8008 Intel Corp. Integrated Rate Matching Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 004 Device 005: ID 23a5:3379 USB Disk 3.0
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 003 Device 003: ID c0f4:01f5 Usb KeyBoard Usb KeyBoard
Bus 003 Device 005: ID 248a:8363 Maxxter Wireless Receiver
Bus 003 Device 002: ID 23a5:3379 USB Disk 3.0
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

> ~ lsusb -t     
/:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 5000M
    |__ Port 2: Dev 5, If 0, Class=Mass Storage, Driver=usb-storage, 5000M
/:  Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/10p, 480M
    |__ Port 1: Dev 2, If 0, Class=Mass Storage, Driver=usb-storage, 480M
    |__ Port 3: Dev 5, If 0, Class=Human Interface Device, Driver=usbhid, 12M
    |__ Port 3: Dev 5, If 1, Class=Human Interface Device, Driver=usbhid, 12M
    |__ Port 4: Dev 3, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
    |__ Port 4: Dev 3, If 1, Class=Human Interface Device, Driver=usbhid, 1.5M
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/2p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/6p, 480M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/2p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M

I did a real world test using Dolphin and monitored with iotop. I transferred 105GB from the internal SATA HDD to the flash drive. While it does now incorporate short bursts up to 30MB/s, it mostly remains at 300KB-5MB/s. The occasional bursts do increase the amount transferred, but this is barely a bandaid solution at best. I stopped the test after 1 hour (judging by the clock 2:25 to 3:25AM) and it has moved 30.3GB of 105GB which averages out to about 8MB/s. Still a lousy number, about 25% of the actual capable speed when accounting for overhead.

However, The initial dump of files transferred at almost quickly up to around 17GB.
The first 17GB was transferred within the first 20 minutes
Between 17 and 22 it started to slow down
The remaining 13GB took 40min.
Perhaps this will give clues?
This would average out to about 14.5MB/s for the first 20 minutes, then ~5MB/s for the remaining 40 minutes which once again, is not normal speeds.

I am going to reboot, and boot the LiveISO on the 2.0 port on the front of the machine and use the 3.0 port that the LiveISO occupied with the USB 3.0 flash drive instead, re add the udev rules list, and rerun these tests.

I can retry rsync and the external 3.0 HDD as well using this method once I move the LiveISO drive.

Hopefully this is enough proof so far to show that there is indeed a problem, but i will retry with it on the 3.0 slot to make sure.

Thanks everyone!

linux-aarhus · 26 August 2023 08:51

I never said it wasn’t a problem …

The main problem is users errornously expecting Linux === Windows.

USB ports on the same controller shares power and bandwidth, which - in case of two devices - roughly would result in a halved transfer rate.

That is not a kernel issue …

And furthermore - specifications rarely matches realworld …

zippydippy · 26 August 2023 08:55

beside all the software-related issues. i’m sure that the external device isn’t designed for this and the thermal stress adds additional file-transfer errors that the system has to correct. this correction-procedures (rewriting/syncing) drops the transfer-rate too.

Again, not trying to be rude, but this is an excuse against real world tests.

It works fine under windows at the correct speed and transfers all the data and can even be safely removed no problem, and so far has worked when running dd with the udev rules given by linux-aarhus, but not in a real test on linux transferring actual files which leads me to believe it’s a bug.

The only one that really stands out as really bad … 18m25s to copy 50G and then 6m24s to unmount - that is the Kingston DataTraveler 64G

My point exactly. I pulled that drive up. It is a USB 3.x (Not type C) Flash drive. On average, your drive is operating at 50GB(50000) divided by 25min (18.5+6.5) for about 33MB/s. It’s a rough estimate, but you see what I am getting at.

Those are not 3.0 speeds, it should be hitting above 80-150MB/s. I’m betting when I restart and use my flash drive in the 3.0 ports properly it will write at that speed like yours, about 33MB/s, which is still incredibly low, but better than 5MB/s.

Thanks for the extra data!

zippydippy · 26 August 2023 09:01

The main problem is users errornously expecting Linux === Windows.

True, but this has been an issue for almost two decades now

USB ports on the same controller shares power and data, which - in case of two devices - roughly would result in a halved transfer rate.

I will redo the test with the drives in different positions leaving the 3.0 port open for the 3.0 flash drive only

That is not a kernel issue …

it might be

And furthermore - specifications rarely matches realworld …

That’s understandable.
What is not is less than 20% actual throughput in real world tests when in staged tests using dd it works as intended.

Sorry we are posting in between each other

Mirdarthos · 26 August 2023 09:06

How…How the **** can the software (the kernel is software after all) have any effect on the physical hardware’s capabilities?

And it will always be the way it is. An “issue” as you call it. Or rather, it will always be an issue until there is a way for software to influence hardware capabilities…

Zesko · 26 August 2023 09:10

I didn’t read much here and skip some posts.

Do you use Swap?

Check inxi
inxi --admin --usb --cpu --machine --disk --partitions --swap