Slow USB transfers

megavolt · 29 August 2023 13:54

Test-Script

#!/usr/bin/env bash

set -o errexit  
set -o pipefail 
set -o nounset 

LANG=C

[[ $EUID != "0" ]] && echo "Run as Root!" && exit

REPEAT=3
MNTPNT='/media/WDElements'

trap rmtestfiles EXIT

rmtestfiles() { sync && rm -fv $MNTPNT/*.testfile && sync;}
dropcache() { sync; echo $1 > /proc/sys/vm/drop_caches; }
rTest() { dd if=$1 of=/dev/null bs=$2 count=$3 iflag=nocache ;}
wTest() { dd if=/dev/urandom of=$1 bs=$2 count=$3 conv=fsync oflag=noatime,nocache;}

rmtestfiles

for size in 100M 10M 1M; do

	echo "### Start Testing $size"

	for byte in 16384K 8192K 4096K 2048K 1024K 512K 256K 128K 64K 32K 16K 8K 4K 2K 1K; do

		count=$(echo "$(numfmt --from=iec $size) / $(numfmt --from=iec $byte)" | bc)
		[[ $count -lt 1 ]] && continue
		testfile="${size}_${count}x$byte.testfile"

		dropcache 3
		for nb in $(seq 1 $REPEAT); do
			echo -n "$byte:$count ($nb) Write Speed: "
			wTest $MNTPNT/$testfile $byte $count 2>&1 | tr '\n' ' ' | cut -d' ' -f14,15,16,17
			#dropcache 3
		done

		dropcache 3
		for nb in $(seq 1 $REPEAT); do
			echo -n "$byte:$count ($nb) Read Speed: "
			rTest $MNTPNT/$testfile $byte $count 2>&1 | tr '\n' ' ' | cut -d' ' -f14,15,16,17
			#dropcache 3
		done

	done

	echo "### End Testing $size"

done

If there is something wrong with the script, please tell. I don’t see a problem.

Well I tried to replicate this “USB Flash Benchmark” with bash and got this:

BTRFS and USB3. Meta and Data duplicated on one single HDD. Zstd level 9 and autodefrag enabled. That is my real world setup for personal cold backups right now.

### Start Testing 100M
16384K:6 (1) Write Speed: 1.29287 s, 77.9 MB/s
16384K:6 (2) Write Speed: 1.50744 s, 66.8 MB/s
16384K:6 (3) Write Speed: 1.56608 s, 64.3 MB/s
16384K:6 (1) Read Speed: 0.53011 s, 190 MB/s
16384K:6 (2) Read Speed: 0.507984 s, 198 MB/s
16384K:6 (3) Read Speed: 0.505723 s, 199 MB/s
8192K:12 (1) Write Speed: 0.884262 s, 114 MB/s
8192K:12 (2) Write Speed: 1.36955 s, 73.5 MB/s
8192K:12 (3) Write Speed: 1.19199 s, 84.4 MB/s
8192K:12 (1) Read Speed: 0.556886 s, 181 MB/s
8192K:12 (2) Read Speed: 0.479545 s, 210 MB/s
8192K:12 (3) Read Speed: 0.479772 s, 210 MB/s
4096K:25 (1) Write Speed: 1.29542 s, 80.9 MB/s
4096K:25 (2) Write Speed: 1.30706 s, 80.2 MB/s
4096K:25 (3) Write Speed: 1.70395 s, 61.5 MB/s
4096K:25 (1) Read Speed: 0.488619 s, 215 MB/s
4096K:25 (2) Read Speed: 0.473155 s, 222 MB/s
4096K:25 (3) Read Speed: 0.472469 s, 222 MB/s
2048K:50 (1) Write Speed: 0.876607 s, 120 MB/s
2048K:50 (2) Write Speed: 1.48282 s, 70.7 MB/s
2048K:50 (3) Write Speed: 1.1549 s, 90.8 MB/s
2048K:50 (1) Read Speed: 0.579834 s, 181 MB/s
2048K:50 (2) Read Speed: 0.498794 s, 210 MB/s
2048K:50 (3) Read Speed: 0.49789 s, 211 MB/s
1024K:100 (1) Write Speed: 0.922097 s, 114 MB/s
1024K:100 (2) Write Speed: 1.07112 s, 97.9 MB/s
1024K:100 (3) Write Speed: 1.41727 s, 74.0 MB/s
1024K:100 (1) Read Speed: 0.606878 s, 173 MB/s
1024K:100 (2) Read Speed: 0.486399 s, 216 MB/s
1024K:100 (3) Read Speed: 0.508281 s, 206 MB/s
512K:200 (1) Write Speed: 0.895902 s, 117 MB/s
512K:200 (2) Write Speed: 1.15331 s, 90.9 MB/s
512K:200 (3) Write Speed: 1.68063 s, 62.4 MB/s
512K:200 (1) Read Speed: 0.66513 s, 158 MB/s
512K:200 (2) Read Speed: 0.498464 s, 210 MB/s
512K:200 (3) Read Speed: 0.498793 s, 210 MB/s
256K:400 (1) Write Speed: 0.935493 s, 112 MB/s
256K:400 (2) Write Speed: 1.34326 s, 78.1 MB/s
256K:400 (3) Write Speed: 1.36245 s, 77.0 MB/s
256K:400 (1) Read Speed: 0.814263 s, 129 MB/s
256K:400 (2) Read Speed: 0.516276 s, 203 MB/s
256K:400 (3) Read Speed: 0.523649 s, 200 MB/s
128K:800 (1) Write Speed: 1.35716 s, 77.3 MB/s
128K:800 (2) Write Speed: 1.68334 s, 62.3 MB/s
128K:800 (3) Write Speed: 1.5773 s, 66.5 MB/s
128K:800 (1) Read Speed: 0.902514 s, 116 MB/s
128K:800 (2) Read Speed: 0.473502 s, 221 MB/s
128K:800 (3) Read Speed: 0.473593 s, 221 MB/s
64K:1600 (1) Write Speed: 1.43384 s, 73.1 MB/s
64K:1600 (2) Write Speed: 1.45282 s, 72.2 MB/s
64K:1600 (3) Write Speed: 1.61266 s, 65.0 MB/s
64K:1600 (1) Read Speed: 0.774598 s, 135 MB/s
64K:1600 (2) Read Speed: 0.49754 s, 211 MB/s
64K:1600 (3) Read Speed: 0.482019 s, 218 MB/s
32K:3200 (1) Write Speed: 1.2607 s, 83.2 MB/s
32K:3200 (2) Write Speed: 1.51682 s, 69.1 MB/s
32K:3200 (3) Write Speed: 1.81873 s, 57.7 MB/s
32K:3200 (1) Read Speed: 0.677298 s, 155 MB/s
32K:3200 (2) Read Speed: 0.489835 s, 214 MB/s
32K:3200 (3) Read Speed: 0.490294 s, 214 MB/s
16K:6400 (1) Write Speed: 1.09833 s, 95.5 MB/s
16K:6400 (2) Write Speed: 1.42074 s, 73.8 MB/s
16K:6400 (3) Write Speed: 1.38223 s, 75.9 MB/s
16K:6400 (1) Read Speed: 0.853249 s, 123 MB/s
16K:6400 (2) Read Speed: 0.490273 s, 214 MB/s
16K:6400 (3) Read Speed: 0.48123 s, 218 MB/s
8K:12800 (1) Write Speed: 1.22265 s, 85.8 MB/s
8K:12800 (2) Write Speed: 1.43112 s, 73.3 MB/s
8K:12800 (3) Write Speed: 1.632 s, 64.3 MB/s
8K:12800 (1) Read Speed: 0.653412 s, 160 MB/s
8K:12800 (2) Read Speed: 0.49403 s, 212 MB/s
8K:12800 (3) Read Speed: 0.484151 s, 217 MB/s
4K:25600 (1) Write Speed: 1.46167 s, 71.7 MB/s
4K:25600 (2) Write Speed: 1.58583 s, 66.1 MB/s
4K:25600 (3) Write Speed: 1.70035 s, 61.7 MB/s
4K:25600 (1) Read Speed: 0.761363 s, 138 MB/s
4K:25600 (2) Read Speed: 0.518892 s, 202 MB/s
4K:25600 (3) Read Speed: 0.543795 s, 193 MB/s
2K:51200 (1) Write Speed: 1.08396 s, 96.7 MB/s
2K:51200 (2) Write Speed: 1.30224 s, 80.5 MB/s
2K:51200 (3) Write Speed: 1.91481 s, 54.8 MB/s
2K:51200 (1) Read Speed: 1.66836 s, 62.9 MB/s
2K:51200 (2) Read Speed: 1.35706 s, 77.3 MB/s
2K:51200 (3) Read Speed: 1.72659 s, 60.7 MB/s
1K:102400 (1) Write Speed: 0.978252 s, 107 MB/s
1K:102400 (2) Write Speed: 1.39898 s, 75.0 MB/s
1K:102400 (3) Write Speed: 1.85517 s, 56.5 MB/s
1K:102400 (1) Read Speed: 2.78827 s, 37.6 MB/s
1K:102400 (2) Read Speed: 3.01967 s, 34.7 MB/s
1K:102400 (3) Read Speed: 2.85055 s, 36.8 MB/s
### End Testing 100M
### Start Testing 10M
8192K:1 (1) Write Speed: 0.073653 s, 114 MB/s
8192K:1 (2) Write Speed: 0.140779 s, 59.6 MB/s
8192K:1 (3) Write Speed: 0.153383 s, 54.7 MB/s
8192K:1 (1) Read Speed: 0.0579272 s, 145 MB/s
8192K:1 (2) Read Speed: 0.0563245 s, 149 MB/s
8192K:1 (3) Read Speed: 0.0531636 s, 158 MB/s
4096K:2 (1) Write Speed: 0.0693751 s, 121 MB/s
4096K:2 (2) Write Speed: 0.118055 s, 71.1 MB/s
4096K:2 (3) Write Speed: 0.111455 s, 75.3 MB/s
4096K:2 (1) Read Speed: 0.207045 s, 40.5 MB/s
4096K:2 (2) Read Speed: 0.0484289 s, 173 MB/s
4096K:2 (3) Read Speed: 0.0487514 s, 172 MB/s
2048K:5 (1) Write Speed: 0.0688738 s, 152 MB/s
2048K:5 (2) Write Speed: 0.0679105 s, 154 MB/s
2048K:5 (3) Write Speed: 0.0707048 s, 148 MB/s
2048K:5 (1) Read Speed: 0.0916102 s, 114 MB/s
2048K:5 (2) Read Speed: 0.0514529 s, 204 MB/s
2048K:5 (3) Read Speed: 0.0570046 s, 184 MB/s
1024K:10 (1) Write Speed: 0.0772935 s, 136 MB/s
1024K:10 (2) Write Speed: 0.1185 s, 88.5 MB/s
1024K:10 (3) Write Speed: 0.11514 s, 91.1 MB/s
1024K:10 (1) Read Speed: 0.294325 s, 35.6 MB/s
1024K:10 (2) Read Speed: 0.039932 s, 263 MB/s
1024K:10 (3) Read Speed: 0.0401828 s, 261 MB/s
512K:20 (1) Write Speed: 0.0860447 s, 122 MB/s
512K:20 (2) Write Speed: 0.107296 s, 97.7 MB/s
512K:20 (3) Write Speed: 0.0926687 s, 113 MB/s
512K:20 (1) Read Speed: 0.264115 s, 39.7 MB/s
512K:20 (2) Read Speed: 0.057163 s, 183 MB/s
512K:20 (3) Read Speed: 0.0569137 s, 184 MB/s
256K:40 (1) Write Speed: 0.135967 s, 77.1 MB/s
256K:40 (2) Write Speed: 0.117229 s, 89.4 MB/s
256K:40 (3) Write Speed: 0.0851557 s, 123 MB/s
256K:40 (1) Read Speed: 0.0837775 s, 125 MB/s
256K:40 (2) Read Speed: 0.0529656 s, 198 MB/s
256K:40 (3) Read Speed: 0.0568389 s, 184 MB/s
128K:80 (1) Write Speed: 0.118512 s, 88.5 MB/s
128K:80 (2) Write Speed: 0.128081 s, 81.9 MB/s
128K:80 (3) Write Speed: 0.116963 s, 89.7 MB/s
128K:80 (1) Read Speed: 0.132672 s, 79.0 MB/s
128K:80 (2) Read Speed: 0.0340472 s, 308 MB/s
128K:80 (3) Read Speed: 0.0328481 s, 319 MB/s
64K:160 (1) Write Speed: 0.0775761 s, 135 MB/s
64K:160 (2) Write Speed: 0.260987 s, 40.2 MB/s
64K:160 (3) Write Speed: 0.123599 s, 84.8 MB/s
64K:160 (1) Read Speed: 0.118175 s, 88.7 MB/s
64K:160 (2) Read Speed: 0.0405875 s, 258 MB/s
64K:160 (3) Read Speed: 0.0404779 s, 259 MB/s
32K:320 (1) Write Speed: 0.410012 s, 25.6 MB/s
32K:320 (2) Write Speed: 0.0824715 s, 127 MB/s
32K:320 (3) Write Speed: 0.100414 s, 104 MB/s
32K:320 (1) Read Speed: 0.103047 s, 102 MB/s
32K:320 (2) Read Speed: 0.0459528 s, 228 MB/s
32K:320 (3) Read Speed: 0.0402611 s, 260 MB/s
16K:640 (1) Write Speed: 0.120041 s, 87.4 MB/s
16K:640 (2) Write Speed: 0.125312 s, 83.7 MB/s
16K:640 (3) Write Speed: 0.086437 s, 121 MB/s
16K:640 (1) Read Speed: 0.183308 s, 57.2 MB/s
16K:640 (2) Read Speed: 0.0507393 s, 207 MB/s
16K:640 (3) Read Speed: 0.0568081 s, 185 MB/s
8K:1280 (1) Write Speed: 0.114561 s, 91.5 MB/s
8K:1280 (2) Write Speed: 0.375556 s, 27.9 MB/s
8K:1280 (3) Write Speed: 0.102337 s, 102 MB/s
8K:1280 (1) Read Speed: 0.127409 s, 82.3 MB/s
8K:1280 (2) Read Speed: 0.0582529 s, 180 MB/s
8K:1280 (3) Read Speed: 0.0571089 s, 184 MB/s
4K:2560 (1) Write Speed: 0.093905 s, 112 MB/s
4K:2560 (2) Write Speed: 0.12697 s, 82.6 MB/s
4K:2560 (3) Write Speed: 0.101461 s, 103 MB/s
4K:2560 (1) Read Speed: 0.121436 s, 86.3 MB/s
4K:2560 (2) Read Speed: 0.0598876 s, 175 MB/s
4K:2560 (3) Read Speed: 0.0651012 s, 161 MB/s
2K:5120 (1) Write Speed: 0.0973148 s, 108 MB/s
2K:5120 (2) Write Speed: 0.158308 s, 66.2 MB/s
2K:5120 (3) Write Speed: 0.0837571 s, 125 MB/s
2K:5120 (1) Read Speed: 0.152488 s, 68.8 MB/s
2K:5120 (2) Read Speed: 0.0738526 s, 142 MB/s
2K:5120 (3) Read Speed: 0.0738581 s, 142 MB/s
1K:10240 (1) Write Speed: 0.126738 s, 82.7 MB/s
1K:10240 (2) Write Speed: 0.198501 s, 52.8 MB/s
1K:10240 (3) Write Speed: 0.186592 s, 56.2 MB/s
1K:10240 (1) Read Speed: 0.132547 s, 79.1 MB/s
1K:10240 (2) Read Speed: 0.0910193 s, 115 MB/s
1K:10240 (3) Read Speed: 0.0660314 s, 159 MB/s
### End Testing 10M
### Start Testing 1M
1024K:1 (1) Write Speed: 0.0122246 s, 85.8 MB/s
1024K:1 (2) Write Speed: 0.00987548 s, 106 MB/s
1024K:1 (3) Write Speed: 0.0103183 s, 102 MB/s
1024K:1 (1) Read Speed: 0.0215833 s, 48.6 MB/s
1024K:1 (2) Read Speed: 0.00514755 s, 204 MB/s
1024K:1 (3) Read Speed: 0.0059678 s, 176 MB/s
512K:2 (1) Write Speed: 0.0163474 s, 64.1 MB/s
512K:2 (2) Write Speed: 0.0189675 s, 55.3 MB/s
512K:2 (3) Write Speed: 0.0187888 s, 55.8 MB/s
512K:2 (1) Read Speed: 0.0195013 s, 53.8 MB/s
512K:2 (2) Read Speed: 0.00379879 s, 276 MB/s
512K:2 (3) Read Speed: 0.00388944 s, 270 MB/s
256K:4 (1) Write Speed: 0.00933148 s, 112 MB/s
256K:4 (2) Write Speed: 0.00932906 s, 112 MB/s
256K:4 (3) Write Speed: 0.00925712 s, 113 MB/s
256K:4 (1) Read Speed: 0.0357854 s, 29.3 MB/s
256K:4 (2) Read Speed: 0.0132441 s, 79.2 MB/s
256K:4 (3) Read Speed: 0.0130606 s, 80.3 MB/s
128K:8 (1) Write Speed: 0.00976928 s, 107 MB/s
128K:8 (2) Write Speed: 0.0109926 s, 95.4 MB/s
128K:8 (3) Write Speed: 0.010884 s, 96.3 MB/s
128K:8 (1) Read Speed: 0.0558117 s, 18.8 MB/s
128K:8 (2) Read Speed: 0.00449468 s, 233 MB/s
128K:8 (3) Read Speed: 0.0041435 s, 253 MB/s
64K:16 (1) Write Speed: 0.0100318 s, 105 MB/s
64K:16 (2) Write Speed: 0.0815589 s, 12.9 MB/s
64K:16 (3) Write Speed: 0.0165261 s, 63.4 MB/s
64K:16 (1) Read Speed: 0.0545618 s, 19.2 MB/s
64K:16 (2) Read Speed: 0.00477978 s, 219 MB/s
64K:16 (3) Read Speed: 0.0044795 s, 234 MB/s
32K:32 (1) Write Speed: 0.011762 s, 89.1 MB/s
32K:32 (2) Write Speed: 0.0168813 s, 62.1 MB/s
32K:32 (3) Write Speed: 0.017198 s, 61.0 MB/s
32K:32 (1) Read Speed: 0.0363938 s, 28.8 MB/s
32K:32 (2) Read Speed: 0.0139402 s, 75.2 MB/s
32K:32 (3) Read Speed: 0.0136268 s, 76.9 MB/s
16K:64 (1) Write Speed: 0.0101102 s, 104 MB/s
16K:64 (2) Write Speed: 0.0135958 s, 77.1 MB/s
16K:64 (3) Write Speed: 0.0169257 s, 62.0 MB/s
16K:64 (1) Read Speed: 0.0295203 s, 35.5 MB/s
16K:64 (2) Read Speed: 0.0123436 s, 84.9 MB/s
16K:64 (3) Read Speed: 0.00809175 s, 130 MB/s
8K:128 (1) Write Speed: 0.0109601 s, 95.7 MB/s
8K:128 (2) Write Speed: 0.0103475 s, 101 MB/s
8K:128 (3) Write Speed: 0.0101721 s, 103 MB/s
8K:128 (1) Read Speed: 0.0312201 s, 33.6 MB/s
8K:128 (2) Read Speed: 0.0146523 s, 71.6 MB/s
8K:128 (3) Read Speed: 0.0154254 s, 68.0 MB/s
4K:256 (1) Write Speed: 0.0142551 s, 73.6 MB/s
4K:256 (2) Write Speed: 0.00996405 s, 105 MB/s
4K:256 (3) Write Speed: 0.0101978 s, 103 MB/s
4K:256 (1) Read Speed: 0.0338269 s, 31.0 MB/s
4K:256 (2) Read Speed: 0.0153038 s, 68.5 MB/s
4K:256 (3) Read Speed: 0.00647492 s, 162 MB/s
2K:512 (1) Write Speed: 0.0104947 s, 99.9 MB/s
2K:512 (2) Write Speed: 0.0105306 s, 99.6 MB/s
2K:512 (3) Write Speed: 0.0105412 s, 99.5 MB/s
2K:512 (1) Read Speed: 0.0357558 s, 29.3 MB/s
2K:512 (2) Read Speed: 0.015758 s, 66.5 MB/s
2K:512 (3) Read Speed: 0.0131153 s, 80.0 MB/s
1K:1024 (1) Write Speed: 0.0183438 s, 57.2 MB/s
1K:1024 (2) Write Speed: 0.0157563 s, 66.5 MB/s
1K:1024 (3) Write Speed: 0.0157651 s, 66.5 MB/s
1K:1024 (1) Read Speed: 0.0264294 s, 39.7 MB/s
1K:1024 (2) Read Speed: 0.0135319 s, 77.5 MB/s
1K:1024 (3) Read Speed: 0.0141164 s, 74.3 MB/s
### End Testing 1M

PS: @zippydippy please use code blocks instead of block quotes for logs and terminal outputs. Thanks.

Wollie · 29 August 2023 14:32

You can repeat this hundeds of time - it’s not getting more convincing as all others here seem to get a different result on their systems, rather proving - it’s not a general bug - but linked to your specific situation.

linux-aarhus · 29 August 2023 15:09

That is great - yet you keep stating the Linux kernel is at fault because you cannot get on par with specification.

I doesn’t matter how many times you repeat yourself - if you keep doing it the same way you get the same results - to get different result yo need to adjust the parameters.

You can state what you like but it seems it is an issue local to your conditions.

Something came out it though.

A lot of tests proving that many other members doesn’t experience the same thing.

My personal opinion is that you are figthing windmills.

My personal gain was that my usb-dev-sync rule got an overhaul - with a script and a config file making it possible to tweak the rule on the fly - even disabling applying the bdi tweaks. (https://codeberg.org/wonky/udev-usb-sync)

The @megavolt sparking script testing a SanDisk Extreme 64G USB-3 type A

 $ sudo bash spark-test.sh
### Start Testing 100M
16384K:6 (1) Write Speed: 2,05214 s, 49,1 MB/s
16384K:6 (2) Write Speed: 1,4047 s, 71,7 MB/s
16384K:6 (3) Write Speed: 1,43235 s, 70,3 MB/s
16384K:6 (1) Read Speed: 0,37106 s, 271 MB/s
16384K:6 (2) Read Speed: 0,369969 s, 272 MB/s
16384K:6 (3) Read Speed: 0,368977 s, 273 MB/s
8192K:12 (1) Write Speed: 2,6297 s, 38,3 MB/s
8192K:12 (2) Write Speed: 2,08014 s, 48,4 MB/s
8192K:12 (3) Write Speed: 2,11133 s, 47,7 MB/s
8192K:12 (1) Read Speed: 0,368765 s, 273 MB/s
8192K:12 (2) Read Speed: 0,369788 s, 272 MB/s
8192K:12 (3) Read Speed: 0,367604 s, 274 MB/s
4096K:25 (1) Write Speed: 1,59597 s, 65,7 MB/s
4096K:25 (2) Write Speed: 3,33137 s, 31,5 MB/s
4096K:25 (3) Write Speed: 3,15013 s, 33,3 MB/s
4096K:25 (1) Read Speed: 0,383942 s, 273 MB/s
4096K:25 (2) Read Speed: 0,382876 s, 274 MB/s
4096K:25 (3) Read Speed: 0,383541 s, 273 MB/s
2048K:50 (1) Write Speed: 3,63908 s, 28,8 MB/s
2048K:50 (2) Write Speed: 2,00728 s, 52,2 MB/s
2048K:50 (3) Write Speed: 2,55113 s, 41,1 MB/s
2048K:50 (1) Read Speed: 0,382664 s, 274 MB/s
2048K:50 (2) Read Speed: 0,381818 s, 275 MB/s
2048K:50 (3) Read Speed: 0,382936 s, 274 MB/s
1024K:100 (1) Write Speed: 3,15911 s, 33,2 MB/s
1024K:100 (2) Write Speed: 2,23399 s, 46,9 MB/s
1024K:100 (3) Write Speed: 3,17532 s, 33,0 MB/s
1024K:100 (1) Read Speed: 0,379875 s, 276 MB/s
1024K:100 (2) Read Speed: 0,380059 s, 276 MB/s
1024K:100 (3) Read Speed: 0,381402 s, 275 MB/s
512K:200 (1) Write Speed: 4,53888 s, 23,1 MB/s
512K:200 (2) Write Speed: 2,7542 s, 38,1 MB/s
512K:200 (3) Write Speed: 3,96397 s, 26,5 MB/s
512K:200 (1) Read Speed: 0,38132 s, 275 MB/s
512K:200 (2) Read Speed: 0,380622 s, 275 MB/s
512K:200 (3) Read Speed: 0,380046 s, 276 MB/s
256K:400 (1) Write Speed: 6,32879 s, 16,6 MB/s
256K:400 (2) Write Speed: 2,9507 s, 35,5 MB/s
256K:400 (3) Write Speed: 4,72027 s, 22,2 MB/s
256K:400 (1) Read Speed: 0,380359 s, 276 MB/s
256K:400 (2) Read Speed: 0,380118 s, 276 MB/s
256K:400 (3) Read Speed: 0,379374 s, 276 MB/s
128K:800 (1) Write Speed: 5,71256 s, 18,4 MB/s
128K:800 (2) Write Speed: 4,55659 s, 23,0 MB/s
128K:800 (3) Write Speed: 4,37218 s, 24,0 MB/s
128K:800 (1) Read Speed: 0,380362 s, 276 MB/s
128K:800 (2) Read Speed: 0,379976 s, 276 MB/s
128K:800 (3) Read Speed: 0,37988 s, 276 MB/s
64K:1600 (1) Write Speed: 4,96567 s, 21,1 MB/s
64K:1600 (2) Write Speed: 4,7997 s, 21,8 MB/s
64K:1600 (3) Write Speed: 4,94032 s, 21,2 MB/s
64K:1600 (1) Read Speed: 0,38099 s, 275 MB/s
64K:1600 (2) Read Speed: 0,380408 s, 276 MB/s
64K:1600 (3) Read Speed: 0,380953 s, 275 MB/s
32K:3200 (1) Write Speed: 4,53763 s, 23,1 MB/s
32K:3200 (2) Write Speed: 7,64509 s, 13,7 MB/s
32K:3200 (3) Write Speed: 4,10713 s, 25,5 MB/s
32K:3200 (1) Read Speed: 0,383135 s, 274 MB/s
32K:3200 (2) Read Speed: 0,382063 s, 274 MB/s
32K:3200 (3) Read Speed: 0,382135 s, 274 MB/s
16K:6400 (1) Write Speed: 6,64039 s, 15,8 MB/s
16K:6400 (2) Write Speed: 4,39126 s, 23,9 MB/s
16K:6400 (3) Write Speed: 6,97184 s, 15,0 MB/s
16K:6400 (1) Read Speed: 0,380004 s, 276 MB/s
16K:6400 (2) Read Speed: 0,378669 s, 277 MB/s
16K:6400 (3) Read Speed: 0,378726 s, 277 MB/s
8K:12800 (1) Write Speed: 6,37685 s, 16,4 MB/s
8K:12800 (2) Write Speed: 5,41542 s, 19,4 MB/s
8K:12800 (3) Write Speed: 5,89025 s, 17,8 MB/s
8K:12800 (1) Read Speed: 0,380716 s, 275 MB/s
8K:12800 (2) Read Speed: 0,383535 s, 273 MB/s
8K:12800 (3) Read Speed: 0,379984 s, 276 MB/s
4K:25600 (1) Write Speed: 5,54003 s, 18,9 MB/s
4K:25600 (2) Write Speed: 4,70928 s, 22,3 MB/s
4K:25600 (3) Write Speed: 5,80384 s, 18,1 MB/s
4K:25600 (1) Read Speed: 0,383481 s, 273 MB/s
4K:25600 (2) Read Speed: 0,380953 s, 275 MB/s
4K:25600 (3) Read Speed: 0,380996 s, 275 MB/s
2K:51200 (1) Write Speed: 6,36235 s, 16,5 MB/s
2K:51200 (2) Write Speed: 5,55414 s, 18,9 MB/s
2K:51200 (3) Write Speed: 4,38061 s, 23,9 MB/s
2K:51200 (1) Read Speed: 0,380587 s, 276 MB/s
2K:51200 (2) Read Speed: 0,378933 s, 277 MB/s
2K:51200 (3) Read Speed: 0,379268 s, 276 MB/s
1K:102400 (1) Write Speed: 6,61161 s, 15,9 MB/s
1K:102400 (2) Write Speed: 4,77451 s, 22,0 MB/s
1K:102400 (3) Write Speed: 4,41714 s, 23,7 MB/s
1K:102400 (1) Read Speed: 0,379472 s, 276 MB/s
1K:102400 (2) Read Speed: 0,379561 s, 276 MB/s
1K:102400 (3) Read Speed: 0,380521 s, 276 MB/s
### End Testing 100M
### Start Testing 10M
8192K:1 (1) Write Speed: 0,0739056 s, 114 MB/s
8192K:1 (2) Write Speed: 0,128124 s, 65,5 MB/s
8192K:1 (3) Write Speed: 0,380594 s, 22,0 MB/s
8192K:1 (1) Read Speed: 0,0336912 s, 249 MB/s
8192K:1 (2) Read Speed: 0,0329849 s, 254 MB/s
8192K:1 (3) Read Speed: 0,0329667 s, 254 MB/s
4096K:2 (1) Write Speed: 0,094105 s, 89,1 MB/s
4096K:2 (2) Write Speed: 0,117474 s, 71,4 MB/s
4096K:2 (3) Write Speed: 0,0961407 s, 87,3 MB/s
4096K:2 (1) Read Speed: 0,0336606 s, 249 MB/s
4096K:2 (2) Read Speed: 0,0334631 s, 251 MB/s
4096K:2 (3) Read Speed: 0,0335781 s, 250 MB/s
2048K:5 (1) Write Speed: 0,117831 s, 89,0 MB/s
2048K:5 (2) Write Speed: 0,146692 s, 71,5 MB/s
2048K:5 (3) Write Speed: 0,134352 s, 78,0 MB/s
2048K:5 (1) Read Speed: 0,0415693 s, 252 MB/s
2048K:5 (2) Read Speed: 0,0414707 s, 253 MB/s
2048K:5 (3) Read Speed: 0,0413742 s, 253 MB/s
1024K:10 (1) Write Speed: 0,763516 s, 13,7 MB/s
1024K:10 (2) Write Speed: 0,150056 s, 69,9 MB/s
1024K:10 (3) Write Speed: 0,806918 s, 13,0 MB/s
1024K:10 (1) Read Speed: 0,0402986 s, 260 MB/s
1024K:10 (2) Read Speed: 0,0402169 s, 261 MB/s
1024K:10 (3) Read Speed: 0,0396693 s, 264 MB/s
512K:20 (1) Write Speed: 0,0948957 s, 110 MB/s
512K:20 (2) Write Speed: 1,06075 s, 9,9 MB/s
512K:20 (3) Write Speed: 0,280024 s, 37,4 MB/s
512K:20 (1) Read Speed: 0,0404846 s, 259 MB/s
512K:20 (2) Read Speed: 0,0400206 s, 262 MB/s
512K:20 (3) Read Speed: 0,0400041 s, 262 MB/s
256K:40 (1) Write Speed: 0,193162 s, 54,3 MB/s
256K:40 (2) Write Speed: 0,133422 s, 78,6 MB/s
256K:40 (3) Write Speed: 0,365768 s, 28,7 MB/s
256K:40 (1) Read Speed: 0,0413312 s, 254 MB/s
256K:40 (2) Read Speed: 0,041479 s, 253 MB/s
256K:40 (3) Read Speed: 0,0412015 s, 254 MB/s
128K:80 (1) Write Speed: 0,258746 s, 40,5 MB/s
128K:80 (2) Write Speed: 2,0192 s, 5,2 MB/s
128K:80 (3) Write Speed: 0,214234 s, 48,9 MB/s
128K:80 (1) Read Speed: 0,0398726 s, 263 MB/s
128K:80 (2) Read Speed: 0,039598 s, 265 MB/s
128K:80 (3) Read Speed: 0,0392598 s, 267 MB/s
64K:160 (1) Write Speed: 1,17706 s, 8,9 MB/s
64K:160 (2) Write Speed: 0,24795 s, 42,3 MB/s
64K:160 (3) Write Speed: 0,815932 s, 12,9 MB/s
64K:160 (1) Read Speed: 0,0397771 s, 264 MB/s
64K:160 (2) Read Speed: 0,0394504 s, 266 MB/s
64K:160 (3) Read Speed: 0,0392795 s, 267 MB/s
32K:320 (1) Write Speed: 0,240529 s, 43,6 MB/s
32K:320 (2) Write Speed: 0,222951 s, 47,0 MB/s
32K:320 (3) Write Speed: 0,276238 s, 38,0 MB/s
32K:320 (1) Read Speed: 0,0394619 s, 266 MB/s
32K:320 (2) Read Speed: 0,0391579 s, 268 MB/s
32K:320 (3) Read Speed: 0,0392658 s, 267 MB/s
16K:640 (1) Write Speed: 1,76122 s, 6,0 MB/s
16K:640 (2) Write Speed: 0,197151 s, 53,2 MB/s
16K:640 (3) Write Speed: 0,199711 s, 52,5 MB/s
16K:640 (1) Read Speed: 0,040725 s, 257 MB/s
16K:640 (2) Read Speed: 0,040987 s, 256 MB/s
16K:640 (3) Read Speed: 0,0404683 s, 259 MB/s
8K:1280 (1) Write Speed: 1,08104 s, 9,7 MB/s
8K:1280 (2) Write Speed: 0,316874 s, 33,1 MB/s
8K:1280 (3) Write Speed: 0,214966 s, 48,8 MB/s
8K:1280 (1) Read Speed: 0,0411942 s, 255 MB/s
8K:1280 (2) Read Speed: 0,040923 s, 256 MB/s
8K:1280 (3) Read Speed: 0,0410421 s, 255 MB/s
4K:2560 (1) Write Speed: 0,226199 s, 46,4 MB/s
4K:2560 (2) Write Speed: 1,33785 s, 7,8 MB/s
4K:2560 (3) Write Speed: 0,245165 s, 42,8 MB/s
4K:2560 (1) Read Speed: 0,0403922 s, 260 MB/s
4K:2560 (2) Read Speed: 0,0401145 s, 261 MB/s
4K:2560 (3) Read Speed: 0,0400941 s, 262 MB/s
2K:5120 (1) Write Speed: 0,220367 s, 47,6 MB/s
2K:5120 (2) Write Speed: 0,216435 s, 48,4 MB/s
2K:5120 (3) Write Speed: 0,367215 s, 28,6 MB/s
2K:5120 (1) Read Speed: 0,0407708 s, 257 MB/s
2K:5120 (2) Read Speed: 0,0406349 s, 258 MB/s
2K:5120 (3) Read Speed: 0,0405965 s, 258 MB/s
1K:10240 (1) Write Speed: 0,263054 s, 39,9 MB/s
1K:10240 (2) Write Speed: 0,815977 s, 12,9 MB/s
1K:10240 (3) Write Speed: 1,30183 s, 8,1 MB/s
1K:10240 (1) Read Speed: 0,0404747 s, 259 MB/s
1K:10240 (2) Read Speed: 0,0402187 s, 261 MB/s
1K:10240 (3) Read Speed: 0,0402079 s, 261 MB/s
### End Testing 10M
### Start Testing 1M
1024K:1 (1) Write Speed: 0,0165847 s, 63,2 MB/s
1024K:1 (2) Write Speed: 0,013923 s, 75,3 MB/s
1024K:1 (3) Write Speed: 0,0141871 s, 73,9 MB/s
1024K:1 (1) Read Speed: 0,00579296 s, 181 MB/s
1024K:1 (2) Read Speed: 0,00527809 s, 199 MB/s
1024K:1 (3) Read Speed: 0,00521193 s, 201 MB/s
512K:2 (1) Write Speed: 0,0176055 s, 59,6 MB/s
512K:2 (2) Write Speed: 0,0148897 s, 70,4 MB/s
512K:2 (3) Write Speed: 0,0133489 s, 78,6 MB/s
512K:2 (1) Read Speed: 0,0054434 s, 193 MB/s
512K:2 (2) Read Speed: 0,00505698 s, 207 MB/s
512K:2 (3) Read Speed: 0,00500104 s, 210 MB/s
256K:4 (1) Write Speed: 0,0216921 s, 48,3 MB/s
256K:4 (2) Write Speed: 0,0154735 s, 67,8 MB/s
256K:4 (3) Write Speed: 0,0248586 s, 42,2 MB/s
256K:4 (1) Read Speed: 0,00573757 s, 183 MB/s
256K:4 (2) Read Speed: 0,00541739 s, 194 MB/s
256K:4 (3) Read Speed: 0,00541014 s, 194 MB/s
128K:8 (1) Write Speed: 0,0238333 s, 44,0 MB/s
128K:8 (2) Write Speed: 0,0136482 s, 76,8 MB/s
128K:8 (3) Write Speed: 0,0246775 s, 42,5 MB/s
128K:8 (1) Read Speed: 0,00633855 s, 165 MB/s
128K:8 (2) Read Speed: 0,00573563 s, 183 MB/s
128K:8 (3) Read Speed: 0,00564526 s, 186 MB/s
64K:16 (1) Write Speed: 0,0525838 s, 19,9 MB/s
64K:16 (2) Write Speed: 0,0491831 s, 21,3 MB/s
64K:16 (3) Write Speed: 0,0249114 s, 42,1 MB/s
64K:16 (1) Read Speed: 0,00618803 s, 169 MB/s
64K:16 (2) Read Speed: 0,00593883 s, 177 MB/s
64K:16 (3) Read Speed: 0,0058694 s, 179 MB/s
32K:32 (1) Write Speed: 0,170148 s, 6,2 MB/s
32K:32 (2) Write Speed: 0,0250277 s, 41,9 MB/s
32K:32 (3) Write Speed: 0,0483793 s, 21,7 MB/s
32K:32 (1) Read Speed: 0,00561176 s, 187 MB/s
32K:32 (2) Read Speed: 0,00561594 s, 187 MB/s
32K:32 (3) Read Speed: 0,00553002 s, 190 MB/s
16K:64 (1) Write Speed: 0,0708621 s, 14,8 MB/s
16K:64 (2) Write Speed: 0,0170751 s, 61,4 MB/s
16K:64 (3) Write Speed: 0,0385588 s, 27,2 MB/s
16K:64 (1) Read Speed: 0,0068149 s, 154 MB/s
16K:64 (2) Read Speed: 0,00651241 s, 161 MB/s
16K:64 (3) Read Speed: 0,00646453 s, 162 MB/s
8K:128 (1) Write Speed: 0,0421809 s, 24,9 MB/s
8K:128 (2) Write Speed: 0,0259959 s, 40,3 MB/s
8K:128 (3) Write Speed: 0,0242289 s, 43,3 MB/s
8K:128 (1) Read Speed: 0,0068782 s, 152 MB/s
8K:128 (2) Read Speed: 0,00663165 s, 158 MB/s
8K:128 (3) Read Speed: 0,00663756 s, 158 MB/s
4K:256 (1) Write Speed: 0,0229303 s, 45,7 MB/s
4K:256 (2) Write Speed: 0,0656518 s, 16,0 MB/s
4K:256 (3) Write Speed: 0,0247052 s, 42,4 MB/s
4K:256 (1) Read Speed: 0,00678946 s, 154 MB/s
4K:256 (2) Read Speed: 0,00643482 s, 163 MB/s
4K:256 (3) Read Speed: 0,00653557 s, 160 MB/s
2K:512 (1) Write Speed: 0,0210359 s, 49,8 MB/s
2K:512 (2) Write Speed: 0,0864608 s, 12,1 MB/s
2K:512 (3) Write Speed: 0,0437133 s, 24,0 MB/s
2K:512 (1) Read Speed: 0,00687197 s, 153 MB/s
2K:512 (2) Read Speed: 0,0066213 s, 158 MB/s
2K:512 (3) Read Speed: 0,00649175 s, 162 MB/s
1K:1024 (1) Write Speed: 0,347115 s, 3,0 MB/s
1K:1024 (2) Write Speed: 0,711977 s, 1,5 MB/s
1K:1024 (3) Write Speed: 0,0687417 s, 15,3 MB/s
1K:1024 (1) Read Speed: 0,00690923 s, 152 MB/s
1K:1024 (2) Read Speed: 0,00658366 s, 159 MB/s
1K:1024 (3) Read Speed: 0,00651718 s, 161 MB/s
### End Testing 1M
removed '/mnt/test/100M_100x1024K.testfile'
removed '/mnt/test/100M_102400x1K.testfile'
removed '/mnt/test/100M_12800x8K.testfile'
removed '/mnt/test/100M_12x8192K.testfile'
removed '/mnt/test/100M_1600x64K.testfile'
removed '/mnt/test/100M_200x512K.testfile'
removed '/mnt/test/100M_25600x4K.testfile'
removed '/mnt/test/100M_25x4096K.testfile'
removed '/mnt/test/100M_3200x32K.testfile'
removed '/mnt/test/100M_400x256K.testfile'
removed '/mnt/test/100M_50x2048K.testfile'
removed '/mnt/test/100M_51200x2K.testfile'
removed '/mnt/test/100M_6400x16K.testfile'
removed '/mnt/test/100M_6x16384K.testfile'
removed '/mnt/test/100M_800x128K.testfile'
removed '/mnt/test/10M_10240x1K.testfile'
removed '/mnt/test/10M_10x1024K.testfile'
removed '/mnt/test/10M_1280x8K.testfile'
removed '/mnt/test/10M_160x64K.testfile'
removed '/mnt/test/10M_1x8192K.testfile'
removed '/mnt/test/10M_20x512K.testfile'
removed '/mnt/test/10M_2560x4K.testfile'
removed '/mnt/test/10M_2x4096K.testfile'
removed '/mnt/test/10M_320x32K.testfile'
removed '/mnt/test/10M_40x256K.testfile'
removed '/mnt/test/10M_5120x2K.testfile'
removed '/mnt/test/10M_5x2048K.testfile'
removed '/mnt/test/10M_640x16K.testfile'
removed '/mnt/test/10M_80x128K.testfile'
removed '/mnt/test/1M_1024x1K.testfile'
removed '/mnt/test/1M_128x8K.testfile'
removed '/mnt/test/1M_16x64K.testfile'
removed '/mnt/test/1M_1x1024K.testfile'
removed '/mnt/test/1M_256x4K.testfile'
removed '/mnt/test/1M_2x512K.testfile'
removed '/mnt/test/1M_32x32K.testfile'
removed '/mnt/test/1M_4x256K.testfile'
removed '/mnt/test/1M_512x2K.testfile'
removed '/mnt/test/1M_64x16K.testfile'
removed '/mnt/test/1M_8x128K.testfile'

zippydippy · 30 August 2023 05:31

I was rereading the whole thread decided to post these comments in order to address them as one unit. I realize I have already answered them, perhaps there may be some confusion somewhere.

So, I have posted many benchmark results now. We all have. The results seems normal, the (3.0) drive can be written to, although at a slower rate compared to windows. That’s fine.

Can someone please post an actual data transfer and average out the MB/s to a 3.0 flash drive with an actual folder above 100GB? No, scratch that. Make it 300GB. Just rsync with real data or a transfer via dolphin with a stopwatch on your arm. Whichever is fine by me. And if you have access to an older machine, 2018 and prior, that would be nice. No more benchmark stuff please. None of the SSDs or AMD threadrippers. Surely someone in this forum has access to mechanical hard drives, high capacity USB flash drives, regular hardware, and large data folders?

I have been told repeatedly now that I need to post actual examples, and I have. I have been told that it’s not a bug or generic problem, it clearly is. Just because it doesn’t affect your hardware, doesn’t mean it’s not affecting mine. I have been told its my settings, yet hundreds of other posts online correlate the issue at hand and changing those settings fixes it for some but not others. I have been told that if i keep repeating the same things, I will get the same answers. Not true. Sometimes it works for longer before dropping speeds. Sometimes in smaller transfers, no slowdowns occur at all. I have complied with the benchmark tests, using multiple programs, which only confirms the existence of a bug. I have been told it’s not repeatable, yet I have done it several times on several computers in 5 days. I have asked for someone else to post a test similar to mine, and have received not one… Benchmarks which we have already covered is not where the problem lies. The problem lies in actually transferring files. And please don’t remind me that “you can never get the same result, its based on the size and number of files.” If its below 1MB/s and taking days, something is wrong. It happens in both the terminal and desktop manager. Repeat my test with similar conditions, rather than with benchmarks because benchmarks work fine on my end too after linux correctly recognized the drive on a 3.0 port.

I decided (again) to copy a completely different hard drive in it’s entirety, to another hard drive (USB 3.0) that I dug up out of my back room, on my newer machine. The OS is installed and updated. So it’s completely different hardware. It’s a different OS. I’ve been transferring 289GB currently at a rate of <2MB/s. It is at 118GB, and it has just hit the 12 hour mark. It was at 90GB before I left this morning, which was an hour after I started. So for 11 hours, it has basically done nothing. Completely different machine, EndeavourOS, udev rules added, same result. Better yet, every drive is ext4. So it’s not NTFS, or just a problem with the first machine. I’m watching rsync, it is taking over 4 seconds to transfer 5MB .jpgs from a camera folder.

Please stop telling me you can’t replicate it, and go find a way to replicate it. When you see it happen you’ll know exactly what I’m experiencing and why it’s so frustrating. Search online and see how it goes unanswered on other forums. If I have to dig up more numbers on another machine, so be it. I have another laptop and 2 more desktops. I want to solve this, or have a bug ticket put in somewhere. A 2 hour transfer should not take over 2 days.

Is this post a rant, if I say it is not a rant? Because it’s not supposed to be a rant so much as just laying out the facts that have been ignored while frustrated.

linux-aarhus · 30 August 2023 06:35

I have no USB flashdrives with a capacity over 64G.

And I only have one spinning disk - a Seagate BUP Slim SL 2TB USB 3.0 (A-plug and Micro-B plug)

Depending on the quality of the device and how it is build it the results will be different and it makes no sense to compare apples and oranges.

And why would anyone use a stopwatch when the excellent 'time* app exist?

zippydippy:

CPU:
  Info: model: Intel Core i3-4130 bits: 64 type: MT MCP arch: Haswell
    gen: core 4 level: v3 note: check built: 2013-15 process: Intel 22nm
    family: 6 model-id: 0x3C (60) stepping: 3 microcode: 0x28
  Topology: cpus: 1x cores: 2 tpc: 2 threads: 4 smt: enabled cache:
    L1: 128 KiB desc: d-2x32 KiB; i-2x32 KiB L2: 512 KiB desc: 2x256 KiB
    L3: 3 MiB desc: 1x3 MiB
  Speed (MHz): avg: 3394 high: 3400 min/max: 800/3400 scaling:
    driver: intel_cpufreq governor: schedutil cores: 1: 3393 2: 3392 3: 3400
    4: 3392 bogomips: 27149

I have no systems that age … The closet would be a ThinkPad T550 (?2015) which runs an i7-5600U and 16G RAM - the internal drive is SSD formatted with f2fs.

It has 3 USB ports where one is USB 3 Standard A.

I went through my box of electronics of various kinds and found and found a Samsung ST750ML022 from 2012.

The members who has been interested in the issue has been trying to replicate it with no success at all.

For one that repeatly uses the phrase not to be rude you begin to appear rude - demanding that we agree with your point of view?

If we cannot replicate - then we cannot replicate - and it is what it is for the system tested - whether you like it or not.

We cannot tell you what you want to hear even if you demand it.

It doesn’t matter if it is an older system or new - the usb device is the bottleneck.

I don’t have any USB 3.0 flash stick - the closest I get is my 64G SanDisk.

The only other option I have is an USB 3.0 enclosure with blue A-plug for the system and and micro-B connector for the enclosure. This host an and old OCZ SATA device.

I will do one last test

using T550
Seagate BUP Slim SL 2TB USB 3.0 plugged into USB2
Samsung ST750ML022 ntfs formatted in USB 3.0 enclosure plugged into USB3

Sysinfo

09:29:37 ○ [fh@panther] ~
 $ pacman-mirrors -G
unstable

09:29:52 ○ [fh@panther] ~
 $ inxi -SCm
System:
  Host: panther Kernel: 6.4.12-1-MANJARO arch: x86_64 bits: 64
    Desktop: KDE Plasma v: 5.27.7 Distro: Manjaro Linux
Memory:
  System RAM: total: 16 GiB available: 15.52 GiB used: 1.55 GiB (10.0%)
  RAM Report: permissions: Unable to run dmidecode. Root privileges
    required.
CPU:
  Info: dual core model: Intel Core i7-5600U bits: 64 type: MT MCP cache:
    L2: 512 KiB
  Speed (MHz): avg: 2746 min/max: 500/3200 cores: 1: 3200 2: 2594 3: 2596
    4: 2594

09:30:07 ○ [fh@panther] ~
 $ lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda      8:0    0 476,9G  0 disk
├─sda1   8:1    0   300M  0 part /boot/efi
├─sda2   8:2    0 459,6G  0 part /
└─sda3   8:3    0  17,1G  0 part [SWAP]
sdc      8:32   0   1,8T  0 disk
└─sdc1   8:33   0   1,8T  0 part
sdd      8:48   0 698,6G  0 disk
└─sdd1   8:49   0 698,6G  0 part

pamac build udev-usb-sync
time rsync -av --progress --partial \
        /mnt/seagate-usb/private /mnt/seagate-usb/software \
        /mnt/Seagate750ntfs

09:30:32 ○ [fh@panther] ~
 $ su -l root
Password:
[panther ~]# mount /dev/sdc1 /mnt/seagate-usb/
[panther ~]# mount /dev/sdd1 /mnt/Samsung750ntfs/
[panther ~]# cd /mnt/seagate-usb/
[panther seagate-usb]# echo $(du -h ./private | tail -n 1 | awk '{print $1}' | tr -d '\n')
36G
[panther seagate-usb]# echo $(du -h ./software | tail -n 1 | awk '{print $1}' | tr -d '\n')
242G
[panther seagate-usb]# rm -rf ../Samsung750ntfs/*
[panther seagate-usb]# time rsync -av --progress --partial private software ../Samsung750ntfs/
sending incremental file list

[...]

real    71m57,124s
user    3m45,845s
sys     11m37,127s

EDIT - the test didn’t finish as I had other things on my mind.

The reverted version (but then I need to copy all my backup data to my nas)

using T550
Seagate BUP Slim SL 2TB USB 3.0 plugged into USB3
Samsung ST750ML022 ntfs formatted in USB 3.0 enclosure plugged into USB2

EDIT 2023-08-31T22:00:00Z
Found time to do some more real world copying.

Old Samsung SpinPoint ST750LM022
From Amazon About this item

Samsung SpinPoint ST750LM022 750GB 2.5-inch Hard Drive General Features: 750 GB capacity
SATA interface 3 Gbps transfer rate 5400 RPM spindle speed 8 MB buffer
12 ms Average Seek Time (typical) 5.6 ms Average Latency 4 second Drive Ready Time SilentSeek
NoiseGuard Lead-free 2.5-inch form factor ATA S.M.A.R.T. feature set ATA security mode feature set

11:56:22 ○ [fh@panther] ~
 $ lsblk -f
NAME   FSTYPE FSVER LABEL          UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
sda                                                                                    
├─sda1 vfat   FAT32                8725-B184                             299,1M     0% /boot/efi
├─sda2 f2fs   1.16                 d519cf49-deb0-466c-9cdc-79ff7125cb1d  170,7G    63% /
└─sda3 swap   1     swap           591812b9-9f86-4a22-82ef-850b7624e6fd                [SWAP]
sdb                                                                                    
└─sdb1 ext4   1.0   seagate-usb    b925148a-0729-447a-ad95-4d9e27dcb442  655,6G    59% /mnt/seagate-usb
sdc                                                                                    
└─sdc1 ntfs         samsung750ntfs 5337D78206CA8C03                      698,5G     0% /mnt/Samsung750ntfs

Folder with thousands of files from tiny sources files to huge zip archives and isos. Sizes range from a few kb to 10GB

The size as reported by du

11:59:33 ○ [fh@panther] ~
 $ echo $(du -h ./test | tail -n 1 | awk '{print $1}' | tr -d '\n')
278G

From local ssd (f2fs) to 5400RPM Seagate BUP Slim 2TB (5400rpm) (ext4) connected to USB-3 port transferred using rsync

12:01:34 ○ [fh@panther] ~
 $ time rsync -av ./test/. /mnt/seagate-usb/
sending incremental file list

[...]

sent 297.219.367.236 bytes  received 723.841 bytes  99.321.667,86 bytes/sec
total size is 297.143.890.573  speedup is 1,00

real    49m51,609s
user    8m22,617s
sys     23m49,808s

From local ssd (f2fs) to Samsung ST750ML022 (5400rpm) (ntfs) connected to USB-3 port using USB-3 enclosure transferred using rsync

13:21:00 ○ [fh@panther] ~
 $ time rsync -av ./test/. /mnt/Samsung750ntfs/
sending incremental file list

[...]

sent 297.219.367.236 bytes  received 724.737 bytes  73.469.309,60 bytes/sec
total size is 297.143.890.573  speedup is 1,00

real    67m25,333s
user    3m24,771s
sys     10m53,567s

Olli · 30 August 2023 07:55

no, the bandwith is restricted before the usb-device runs into its limits. i’m still testing but there are restrictions from the scheduler and the handling of priorities of a running process that drop the speed of large file-transfer-copying and these are independent of the external usb-device.
the actual improvements like changing the dirty-bytes do only effect if a system with a swap is used, they don’t effect if there isn’t a swap-device active. disabling the flush is only cosmetic because the data is only stored till unmounting and have to be transferred before the device gets unmounted.
i can confirm that the testing device i’m using (seagate-bk, USB-2, ntfs) is able to copy data with a constant 18,9 MB/s and it reaches 32MB/s in sum if i copy multiple files BUT it always drops the speed on a large-single-file-transfer to ~one-third of the max-transfer-rate that it is capable.
Why ?

linux-aarhus · 30 August 2023 08:02

Interesting … especially

This makes sense as the Manjaro system is not specifically targeting a fileserver.

One Windows configuration comes to mind - the option to prioritize background processes - which is server like in contrast to a desktop which is often prioritizing the active window.

I don’t know if this has any bearing

ulimit -a

This tells me max open files is 8192

Olli · 30 August 2023 12:32

actual speedrecord

dd if=/dev/urandom of=/run/media/OLLI/SeagateSlimDrive/TEST/test_001.txt status=progress
14377754624 Bytes (14 GB, 13 GiB) kopiert, 804 s, 17,9 MB/s^C
28113248+0 Datensätze ein
28113248+0 Datensätze aus
14393982976 Bytes (14 GB, 13 GiB) kopiert, 804,991 s, 17,9 MB/s

there are a lot of things that should and must get better configured in linux. my latest result (speedrecord) at the testing-setup
14GB @ 17,9 MB/s

throtteling cpu’s and especially the governor-control are the hinges but not this old test-drive. i will report more later …

p.s.: but with this at the beginning

dd if=/dev/urandom of=/run/media/OLLI/SeagateSlimDrive/TEST/test_005.txt status=progress                                                                     
4834000896 Bytes (4,8 GB, 4,5 GiB) kopiert, 804 s, 6,0 MB/s^C
9446147+0 Datensätze ein
9446147+0 Datensätze aus
4836427264 Bytes (4,8 GB, 4,5 GiB) kopiert, 804,864 s, 6,0 MB/s

now it’s 3 times (!) faster and there is no (!) change in the hardware-setup.
the critics of the TO are in fact real

megavolt · 30 August 2023 12:32

HDD yes (I showed it already), Flash Drive no. I found a USB2.0 with 120GB in my junk box, but we’re talking about 7-10MB/s write speed and 14-20MB/s read speed. For small files it is fine and it data can grow with time, still there is data on it. Other Flash Drives are only 4/8/16GB and I never thought about a higher capacity, because flash drives are only reliable to share data (if no internet connection available), but not backup data.

With my HDDs connected by USB3, I cannot reproduce a speed drop after 100GB or more.

If that is the case then I can only imagine that the Hardware, so the Flash Drive (FD) is the culprit here. Or do you talk about Hard Drive Disk (HDD)?

Please distinguish between FD’s and HDDs, since they are completely different.

File fragmentation reduces transfer speed, because the write process get delayed plus it uses already “Extent Allocation” and “Delayed Allocation” and “Online-Defragmentation”. So in general it is a bit slower because of that: Linux prefers defragmented data over speed unlike Windows. Windows uses also “Extent Allocation”, but it doesn’t try defragment online while writing, unlike Linux. Also “Extent Allocation” is implemented differently in Windows and Linux: While Linux randomize the position (Cluster) of the file, Windows follow policies.

This process is known to me that it drops speed when the disk has low free space or high fragmented data on HDD’s. The I/O goes high, but the speed drops. That’s because it defragment while it writes to my knowledge. FD’s are also known to have a low I/O operations throughput.

If heat or a bad quality chip (FD’s) is not the problem, then this might be the root cause.

That is the only issue which can survive 20y+, which you experience here. I can’t imagine anything else. Either way, it is designed that way…

Mirdarthos · 30 August 2023 13:01

FWIW:

This thread is, to me at least, an extremely good example of why USB flash drives aren’t suitable for LARGE quantities of schtuff.

I have a 256GB here (I think that’s the size but can’t be sure,) which I’ve ever only used for my Manjaro Live environment. As @megavolt said, that’s why there’s internet, or when I still had to go into the office, I used an external HDD (which was only 160GB anyway.) And no, I don’t know why anyone would even make that big a drive. probably because they can. So they did. But something a lot of people don’t realize is, just because you can do something, doesn’t mean you should do it…

(This is just an opinion, nothing else.) I highly doubt Flash Drives were originally invented for such large capacities so using them like this is not what they’re designed for…and, no offense, but if you’re trying to do something it’s not designed for, you can’t really complain that it’s not working as it should. (Once again, this is all just opinion and not based on fact.)

Olli · 30 August 2023 13:03

@Mirdarthos

i’m testing on a seagate-usb-hdd not an flash-drive.

the restricting rule is that the file-system has to be ntfs as the TO already stated: he needs ntfs to be universal to any other computer it will also plugged in.

Mirdarthos · 30 August 2023 13:08

My apologies, then. I must’ve lost it somewhere in the 50+ replies.

Olli · 30 August 2023 13:10

i’m cured on any other backup-media. it must be external. once i used a second drive in my computer as backup-drive until the secondary-drive-controller failed and burned the drive to electronic-trash.

Mirdarthos · 30 August 2023 13:13

Yeah.

That’s why I keep my OS backups on a local drive and all documents and images off-site.

Edit:

That’s in addition to splitting my OS from my data…then I can easily un-mount them to do stuff that can be dangerous. It’s just safer this way. _{Not to mention cheaper.}

Olli · 30 August 2023 13:18

i agree but external-usb-storage is a very common way for most of the average users. better solutions as NAS etc… do exist, but the average-user would never think about such a solution and in fact it starts at the financial aspect and ends with the technical setup. usb-storage is easy, mobile and in comparison to other ones very valuable at the price-tag.

freggel.doe · 30 August 2023 13:22

Little OT, but for this use case exFat filesystem seems to be a better fit:

Mirdarthos · 30 August 2023 13:22

True true. Most would say much cheaper. And they’d also change their tune when the sh17 hits the fan…

Olli · 30 August 2023 13:26

did you tested it ? i’m already thinking about this too but it’s proven that there are a lot of other settings in linux end up in a very poorish performance on large-usb-file-transfer. i wouldn’t change my test-suite actually by start mixing different file-systems now. maybe later.

freggel.doe · 30 August 2023 13:32

Not in the context of this thread (usb transfer speeds). It does have less overhead than ntfs though which might result in a different experience speed wise.
I’m using it on my go-to usb stick to get photos to the printer in the shop for years.

Olli · 30 August 2023 13:32

you nailed something i’m not aware if you thought about. according to the “speedrecord-result” i posted above:
a lot of thermal- and governor- control that comes with the kernel is time-and performance wasting crap and testing on an imac (that always lags because the apple-design drives temps to max) i was stunned that there is a better thermal- and performance by disabling a lot of it and create a manual configuration, but i will report later.