USB transfer data drops drastically and maximum number of clients are reached

Sr_J · 4 December 2021 12:04

Hello everyone,

In the last months, I had been experiencing some issues whenever I want to transfer data into different USB sticks:

For files with size>1GB, the speed transfer to the USB drive just drops to nearly zero whenever the process reach the first 900MB (more and less). Then, it became very unstable and sometimes it oscillates between “normal” data transfer speed and nearly zero.
At this point, my CPU and memory go high, and sometimes I can no open new desktop apps because a Maximum number of clients reached error.

I have tried to suspend baloo indexing, but apparently nothing changes.

Some information about my ports:

$ lsusb -t
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/6p, 10000M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/12p, 480M
    |__ Port 4: Dev 7, If 0, Class=Mass Storage, Driver=usb-storage, 480M
    |__ Port 8: Dev 3, If 1, Class=Video, Driver=uvcvideo, 480M
    |__ Port 8: Dev 3, If 0, Class=Video, Driver=uvcvideo, 480M
    |__ Port 10: Dev 4, If 0, Class=Wireless, Driver=btusb, 12M
    |__ Port 10: Dev 4, If 1, Class=Wireless, Driver=btusb, 12M

and my system info:

$ inxi -Fazy
System:
  Kernel: 5.10.79-1-MANJARO x86_64 bits: 64 compiler: gcc v: 11.1.0
  parameters: BOOT_IMAGE=/boot/vmlinuz-5.10-x86_64
  root=UUID=3d3d2d93-73e4-464b-b6fd-9723814875e6 rw quiet apparmor=1
  security=apparmor resume=UUID=f15dd4eb-897b-4f69-8365-8e3b5b64fecb
  udev.log_priority=3
  Desktop: KDE Plasma 5.23.3 tk: Qt 5.15.2 wm: kwin_x11 vt: 1 dm: SDDM
  Distro: Manjaro Linux base: Arch Linux
Machine:
  Type: Laptop System: LENOVO product: 20RA001BSP v: ThinkPad E14
  serial: <superuser required> Chassis: type: 10 serial: <superuser required>
  Mobo: LENOVO model: 20RA001BSP v: SDK0J40697 WIN
  serial: <superuser required> UEFI: LENOVO v: R16ET32W (1.18 )
  date: 06/30/2021
Battery:
  ID-1: BAT0 charge: 41.6 Wh (100.0%) condition: 41.6/45.7 Wh (91.0%)
  volts: 12.3 min: 11.1 model: SMP 5B10W138 type: Li-poly serial: <filter>
  status: Full cycles: 120
CPU:
  Info: Quad Core model: Intel Core i7-10510U bits: 64 type: MT MCP
  arch: Kaby Lake note: check family: 6 model-id: 8E (142) stepping: C (12)
  microcode: EA cache: L1: 256 KiB L2: 1024 KiB L3: 8 MiB
  flags: avx avx2 ht lm nx pae sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx
  bogomips: 36812
  Speed: 1607 MHz min/max: 400/4900 MHz Core speeds (MHz): 1: 1607 2: 1784
  3: 1586 4: 1919 5: 1860 6: 2147 7: 3400 8: 3400
  Vulnerabilities: Type: itlb_multihit status: KVM: VMX disabled
  Type: l1tf status: Not affected
  Type: mds status: Not affected
  Type: meltdown status: Not affected
  Type: spec_store_bypass
  mitigation: Speculative Store Bypass disabled via prctl and seccomp
  Type: spectre_v1
  mitigation: usercopy/swapgs barriers and __user pointer sanitization
  Type: spectre_v2 mitigation: Enhanced IBRS, IBPB: conditional, RSB filling
  Type: srbds mitigation: TSX disabled
  Type: tsx_async_abort status: Not affected
Graphics:
  Device-1: Intel CometLake-U GT2 [UHD Graphics] vendor: Lenovo driver: i915
  v: kernel bus-ID: 00:02.0 chip-ID: 8086:9b41 class-ID: 0300
  Device-2: Chicony Integrated Camera type: USB driver: uvcvideo bus-ID: 1-8:3
  chip-ID: 04f2:b6d9 class-ID: 0e02 serial: <filter>
  Display: x11 server: X.Org 1.21.1.1 compositor: kwin_x11 driver:
  loaded: modesetting alternate: fbdev,vesa display-ID: :0 screens: 1
  Screen-1: 0 s-res: 1920x1080 s-dpi: 96 s-size: 508x285mm (20.0x11.2")
  s-diag: 582mm (22.9")
  Monitor-1: eDP-1 res: 1920x1080 hz: 60 dpi: 158 size: 309x173mm (12.2x6.8")
  diag: 354mm (13.9")
  OpenGL: renderer: Mesa Intel UHD Graphics (CML GT2) v: 4.6 Mesa 21.2.5
  direct render: Yes
Audio:
  Device-1: Intel Comet Lake PCH-LP cAVS vendor: Lenovo driver: snd_hda_intel
  v: kernel alternate: snd_soc_skl,snd_sof_pci bus-ID: 00:1f.3
  chip-ID: 8086:02c8 class-ID: 0403
  Sound Server-1: ALSA v: k5.10.79-1-MANJARO running: yes
  Sound Server-2: sndio v: N/A running: no
  Sound Server-3: JACK v: 1.9.19 running: no
  Sound Server-4: PulseAudio v: 15.0 running: yes
  Sound Server-5: PipeWire v: 0.3.40 running: yes
Network:
  Device-1: Intel Comet Lake PCH-LP CNVi WiFi driver: iwlwifi v: kernel
  bus-ID: 00:14.3 chip-ID: 8086:02f0 class-ID: 0280
  IF: wlp0s20f3 state: up mac: <filter>
  Device-2: Realtek RTL8111/8168/8411 PCI Express Gigabit Ethernet
  vendor: Lenovo driver: r8168 v: 8.049.02-NAPI modules: r8169 port: 2000
  bus-ID: 04:00.0 chip-ID: 10ec:8168 class-ID: 0200
  IF: enp4s0 state: down mac: <filter>
Bluetooth:
  Device-1: Intel AX201 Bluetooth type: USB driver: btusb v: 0.8
  bus-ID: 1-10:4 chip-ID: 8087:0026 class-ID: e001
  Report: rfkill ID: hci0 rfk-id: 2 state: up address: see --recommends
Drives:
  Local Storage: total: 480.66 GiB used: 278.4 GiB (57.9%)
  SMART Message: Unable to run smartctl. Root privileges required.
  ID-1: /dev/nvme0n1 maj-min: 259:0 vendor: Samsung model: MZALQ512HALU-000L1
  size: 476.94 GiB block-size: physical: 512 B logical: 512 B speed: 31.6 Gb/s
  lanes: 4 type: SSD serial: <filter> rev: 9L1QFXV7 temp: 21.9 C scheme: GPT
  ID-2: /dev/sda maj-min: 8:0 type: USB model: 090c 1000 size: 3.72 GiB
  block-size: physical: 512 B logical: 512 B type: N/A serial: <filter>
  scheme: MBR
  SMART Message: Unknown USB bridge. Flash drive/Unsupported enclosure?
Partition:
  ID-1: / raw-size: 100 GiB size: 97.87 GiB (97.87%) used: 56.44 GiB (57.7%)
  fs: ext4 dev: /dev/nvme0n1p7 maj-min: 259:7
  ID-2: /boot/efi raw-size: 512 MiB size: 511 MiB (99.80%)
  used: 300 KiB (0.1%) fs: vfat dev: /dev/nvme0n1p5 maj-min: 259:5
  ID-3: /home raw-size: 242.98 GiB size: 238.11 GiB (97.99%)
  used: 219.71 GiB (92.3%) fs: ext4 dev: /dev/nvme0n1p8 maj-min: 259:8
Swap:
  Kernel: swappiness: 60 (default) cache-pressure: 100 (default)
  ID-1: swap-1 type: partition size: 8 GiB used: 784.5 MiB (9.6%) priority: -2
  dev: /dev/nvme0n1p6 maj-min: 259:6
Sensors:
  System Temperatures: cpu: 52.0 C pch: 48.0 C mobo: 0.0 C
  Fan Speeds (RPM): cpu: 0
Info:
  Processes: 455 Uptime: 4h 40m wakeups: 3017 Memory: 15.44 GiB
  used: 11.83 GiB (76.6%) Init: systemd v: 249 tool: systemctl Compilers:
  gcc: 11.1.0 Packages: pacman: 2057 lib: 463 flatpak: 0 Shell: Bash v: 5.1.8
  running-in: konsole inxi: 3.3.09

Could someone give me any help?

Bests.

linux-aarhus · 4 December 2021 15:37

The tweakings required is unique to every system.

Mangle the dirty_{ratio|bytes} and the vfs_cache_pressure options until you achieve the desired result.

Search the forum - it has been discussed on frequent occasions.

Sr_J · 4 December 2021 17:08

Thank you @linux-aarhus, I going to take a look! I hope this resolves the issue.

Nevertheless, I have my Manjaro system installed from a year now… and this data transfer problem appeared one/two months ago. Weird.

Aragorn · 5 December 2021 06:25

This is normal. USB sticks are not as fast as an SSD, and the bus is slower too. In addition to that, writes are buffered, which is why the first 900 MiB will appear to be transferred easily, but after that point, the bus and the buffers become congested because of the slow storage medium.

Another thing to consider is that USB sticks don’t have any Flash Translation Layer, nor support for TRIM. This means that after a certain number of writes and rewrites, the flash storage layer degrades due to write amplification. There’s very little if anything you can do about that. USB sticks are not made to last forever.

Sr_J · 7 December 2021 22:32

Thank you @linux-aarhus & @Aragorn, I have followed some of the indications discussed in The pernicious USB-stick stall problem.

I have used the script of cscs - maxperfwiz (Gitlab) to set the following values:

vm.swappiness=10
vm.vfs_cache_pressure=75
vm.dirty_expire_centisecs=3000
vm.dirty_writeback_centisecs=1500
vm.min_free_kbytes=117395

However, the values suggested for vm.dirty_background_ratio and vm.dirty_ratio by the script do not work, so I have set instead:

vm.dirty_background_bytes=16777216
vm.dirty_bytes=33554432

following the discussion Decrease dirty bytes for more reliable USB transfer.

Now USB transfer seems to work fine in my computer after testing some examples, since the transfer speed is more and less stable and it does not decay at the end abruptly.

linux-aarhus · 8 December 2021 04:21

So in reality the first answer you got was the solution.

And just as the answer stated - every system is different and you have to mangle the settings to fit your system.

system · 10 December 2021 18:22

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.