High load while copying files

Hello. I have weird problem and I don’t know how to debug or find what is the problem. Every time I copy some files or downloading multiple files from cloud or some program is doing something with disk, my system is tanked and load jumps to 10+ up to 40. It’s nvme disk so why is this happening? Is my disk failing? Thanks for suggestions/explanation.

$ inxi --full --admin --filter --width
  Kernel: 6.6.3-1-MANJARO arch: x86_64 bits: 64 compiler: gcc v: 13.2.1
    clocksource: tsc available: acpi_pm
    parameters: BOOT_IMAGE=/boot/vmlinuz-6.6-x86_64
    root=UUID=75b4a339-3527-4ddf-a52a-6c47b44e6130 rw
  Desktop: Xfce v: 4.18.1 tk: Gtk v: 3.24.36 info: xfce4-panel wm: xfwm
    v: 4.18.0 vt: 7 dm: LightDM v: 1.32.0 Distro: Manjaro Linux base: Arch Linux
  Type: Laptop System: LENOVO product: 81B0 v: Lenovo V330-14IKB
    serial: <superuser required> Chassis: type: 10 v: Lenovo V330-14IKB
    serial: <superuser required>
  Mobo: LENOVO model: LNVNB161216 v: SDK0J40697 WIN
    serial: <superuser required> UEFI: LENOVO v: 6UCN54WW(V4.09) date: 11/23/2018
  ID-1: BAT1 charge: 20.4 Wh (96.7%) condition: 21.1/30.0 Wh (70.5%)
    power: 2.9 W volts: 8.6 min: 7.5 model: 0x53 0x4D 0x50 0x00 0x31 0x42 0x4C
    0x31 0x37 0x4D 0x32 0x50 0x42 type: Li-ion serial: <filter>
    status: charging
  Device-1: hidpp_battery_0 model: Logitech Wireless Keyboard
    serial: <filter> charge: 55% (should be ignored) rechargeable: yes
    status: discharging
  Info: model: Intel Core i5-8250U bits: 64 type: MT MCP arch: Coffee Lake
    gen: core 8 level: v3 note: check built: 2017 process: Intel 14nm family: 6
    model-id: 0x8E (142) stepping: 0xA (10) microcode: 0xF4
  Topology: cpus: 1x cores: 4 tpc: 2 threads: 8 smt: enabled cache:
    L1: 256 KiB desc: d-4x32 KiB; i-4x32 KiB L2: 1024 KiB desc: 4x256 KiB
    L3: 6 MiB desc: 1x6 MiB
  Speed (MHz): avg: 749 high: 800 min/max: 400/3400 scaling:
    driver: intel_pstate governor: powersave cores: 1: 799 2: 800 3: 800 4: 800
    5: 799 6: 800 7: 800 8: 400 bogomips: 28808
  Flags: avx avx2 ht lm nx pae sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx
  Type: gather_data_sampling mitigation: Microcode
  Type: itlb_multihit status: KVM: VMX disabled
  Type: l1tf mitigation: PTE Inversion; VMX: conditional cache flushes, SMT
  Type: mds mitigation: Clear CPU buffers; SMT vulnerable
  Type: meltdown mitigation: PTI
  Type: mmio_stale_data mitigation: Clear CPU buffers; SMT vulnerable
  Type: retbleed mitigation: IBRS
  Type: spec_rstack_overflow status: Not affected
  Type: spec_store_bypass mitigation: Speculative Store Bypass disabled via
  Type: spectre_v1 mitigation: usercopy/swapgs barriers and __user pointer
  Type: spectre_v2 mitigation: IBRS, IBPB: conditional, STIBP: conditional,
    RSB filling, PBRSB-eIBRS: Not affected
  Type: srbds mitigation: Microcode
  Type: tsx_async_abort status: Not affected
  Device-1: Intel UHD Graphics 620 vendor: Lenovo driver: i915 v: kernel
    arch: Gen-9.5 process: Intel 14nm built: 2016-20 ports: active: eDP-1
    empty: DP-1,HDMI-A-1,HDMI-A-2 bus-ID: 00:02.0 chip-ID: 8086:5917
    class-ID: 0300
  Device-2: Chicony Integrated Camera (1280x720@30) driver: uvcvideo
    type: USB rev: 2.0 speed: 480 Mb/s lanes: 1 mode: 2.0 bus-ID: 1-5:2
    chip-ID: 04f2:b604 class-ID: 0e02 serial: <filter>
  Display: x11 server: X.Org v: 21.1.9 compositor: xfwm v: 4.18.0 driver: X:
    loaded: modesetting alternate: fbdev,vesa dri: iris gpu: i915
    display-ID: :0.0 screens: 1
  Screen-1: 0 s-res: 1920x1080 s-dpi: 96 s-size: 508x285mm (20.00x11.22")
    s-diag: 582mm (22.93")
  Monitor-1: eDP-1 model: BOE Display 0x06bb built: 2016 res: 1920x1080
    hz: 60 dpi: 158 gamma: 1.2 size: 309x173mm (12.17x6.81") diag: 354mm (13.9")
    ratio: 16:9 modes: 1920x1080
  API: OpenGL Message: Unable to show GL data. glxinfo is missing.
  Device-1: Intel Sunrise Point-LP HD Audio vendor: Lenovo
    driver: snd_hda_intel v: kernel alternate: snd_soc_skl,snd_soc_avs
    bus-ID: 00:1f.3 chip-ID: 8086:9d71 class-ID: 0403
  API: ALSA v: k6.6.3-1-MANJARO status: kernel-api with: aoss
    type: oss-emulator tools: alsactl,alsamixer,amixer
  Server-1: sndiod v: N/A status: off tools: aucat,midicat,sndioctl
  Server-2: JACK v: 1.9.22 status: off tools: N/A
  Server-3: PipeWire v: 1.0.0 status: off tools: pw-cli
  Server-4: PulseAudio v: 16.1 status: active with: 1: pulseaudio-alsa
    type: plugin 2: pulseaudio-jack type: module tools: pacat,pactl,pavucontrol
  Device-1: Realtek RTL8111/8168/8411 PCI Express Gigabit Ethernet
    vendor: Lenovo driver: r8168 v: 8.052.01-NAPI modules: r8169 pcie: gen: 1
    speed: 2.5 GT/s lanes: 1 port: 4000 bus-ID: 01:00.0 chip-ID: 10ec:8168
    class-ID: 0200
  IF: enp1s0 state: down mac: <filter>
  Device-2: Intel Dual Band Wireless-AC 3165 Plus Bluetooth driver: iwlwifi
    v: kernel pcie: gen: 1 speed: 2.5 GT/s lanes: 1 bus-ID: 02:00.0
    chip-ID: 8086:3166 class-ID: 0280
  IF: wlp2s0 state: up mac: <filter>
  Device-1: Intel Bluetooth wireless interface driver: btusb v: 0.8 type: USB
    rev: 2.0 speed: 12 Mb/s lanes: 1 mode: 1.1 bus-ID: 1-6:3 chip-ID: 8087:0a2a
    class-ID: e001
  Report: btmgmt ID: hci0 rfk-id: 3 state: up address: <filter> bt-v: 4.2
    lmp-v: 8 status: discoverable: no pairing: no class-ID: 7c010c
  Local Storage: total: 238.47 GiB used: 167.58 GiB (70.3%)
  SMART Message: Unable to run smartctl. Root privileges required.
  ID-1: /dev/nvme0n1 maj-min: 259:0 vendor: SK Hynix
    model: HFS256GD9TNG-62A0A size: 238.47 GiB block-size: physical: 512 B
    logical: 512 B speed: 31.6 Gb/s lanes: 4 tech: SSD serial: <filter>
    fw-rev: 80001E00 temp: 55.9 C scheme: GPT
  ID-1: / raw-size: 229.37 GiB size: 224.71 GiB (97.97%)
    used: 167.57 GiB (74.6%) fs: ext4 dev: /dev/dm-0 maj-min: 254:0
    mapped: luks-aa016456-ecb8-4596-89b8-1ca0be5e009e
  ID-2: /boot/efi raw-size: 300 MiB size: 299.4 MiB (99.80%)
    used: 408 KiB (0.1%) fs: vfat dev: /dev/nvme0n1p1 maj-min: 259:1
  Alert: No swap data was found.
  System Temperatures: cpu: 45.0 C pch: 43.0 C mobo: N/A
  Fan Speeds (rpm): N/A
  Processes: 242 Uptime: 11m wakeups: 0 Memory: total: 20 GiB note: est.
  available: 19.28 GiB used: 1.46 GiB (7.6%) Init: systemd v: 254
  default: graphical tool: systemctl Compilers: gcc: 13.2.1 clang: 16.0.6
  Packages: pm: dpkg pkgs: 0 pm: pacman pkgs: 2229 libs: 567
  tools: pacaur,pamac,yay pm: flatpak pkgs: 0 Shell: Bash v: 5.2.21
  running-in: xfce4-terminal inxi: 3.3.31

$ sudo smartctl -a /dev/nvme0
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.6.3-1-MANJARO] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

Model Number:                       HFS256GD9TNG-62A0A
Serial Number:                      EJ84N698211504P3G
Firmware Version:                   80001E00
PCI Vendor/Subsystem ID:            0x1c5c
IEEE OUI Identifier:                0xace42e
Controller ID:                      1
NVMe Version:                       1.2
Number of Namespaces:               1
Namespace 1 Size/Capacity:          256 060 514 304 [256 GB]
Namespace 1 Utilization:            255 810 449 408 [255 GB]
Namespace 1 Formatted LBA Size:     512
Local Time is:                      Wed Dec  6 00:44:21 2023 CET
Firmware Updates (0x14):            2 Slots, no Reset required
Optional Admin Commands (0x0017):   Security Format Frmw_DL Self_Test
Optional NVM Commands (0x001f):     Comp Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat
Log Page Attributes (0x02):         Cmd_Eff_Lg
Maximum Data Transfer Size:         32 Pages
Warning  Comp. Temp. Threshold:     79 Celsius
Critical Comp. Temp. Threshold:     81 Celsius

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +     6.00W       -        -    0  0  0  0        5       5
 1 +     3.80W       -        -    1  1  1  1       30      30
 2 +     2.40W       -        -    2  2  2  2      100     100
 3 -   0.0700W       -        -    3  3  3  3     1000    1000
 4 -   0.0070W       -        -    3  3  3  3     1000    5000

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 +     512       0         0
 1 -    4096       0         0

SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        35 Celsius
Available Spare:                    100%
Available Spare Threshold:          10%
Percentage Used:                    12%
Data Units Read:                    17 335 375 [8,87 TB]
Data Units Written:                 18 222 274 [9,32 TB]
Host Read Commands:                 123 901 075
Host Write Commands:                68 745 221
Controller Busy Time:               19 074
Power Cycles:                       2 135
Power On Hours:                     1 087
Unsafe Shutdowns:                   61
Media and Data Integrity Errors:    0
Error Information Log Entries:      0
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0
Temperature Sensor 1:               35 Celsius
Temperature Sensor 2:               37 Celsius

Error Information (NVMe Log 0x01, 16 of 256 entries)
No Errors Logged

Self-test Log (NVMe Log 0x06)
Self-test status: No self-test in progress
Num  Test_Description  Status                       Power_on_Hours  Failing_LBA  NSID Seg SCT Code
 0   Short             Completed without error                1087            -     -   -   -    -

Hello @pantas.aspro :wink:

No the nvme is just fine.

Note that when writing to a disk, and you have full disk encryption, then it needs to encrypt the data when writing. That results in a higher CPU load, even if your CPU has the aes flag, just much reduced as it would be without.

You have a full disk luks encryption:

Is there a kernel process that runs full on one core? Do you see a process called like “kworker/u16:4+flush” or some similar kworker process? You might want to use a process viewer that can show kernel processes.

Do you see this behavior with LTS Kernel 6.1 or some Kernel version smaller than 6.5?

I thought it might be problem but I have conky running on desktop and I see that CPU is running almost idle around up to max 10%, not one core fully running.
I see kworker process pops up form time to time. Not exactly one you mentioned. But those are usually taking ~1% of cpu power then moves down/disappear.
I have this problem for a long time even on 6.1 kernel, I’ve just started to copy some files lately more and I got fed up.