High CPU usage from udev

Support
1

Hey,
I know that similar topics already exist but since I did not have this problem the first two years with Manjaro and without Hardware or majoar software changes it now just constantly keeps coming up I thought I’d ask here.

So what happens is, that I get an unusually high cpu usage from (udev-worker), when I enter
udevadm monitor
I get that it is constantly recognizing a device change:

KERNEL[760.644667] change   /devices/pci0000:00/0000:00:17.0/ata1/host0/target0:0:0/0:0:0:0/block/sda/sda1 (block)
UDEV  [760.664992] change   /devices/pci0000:00/0000:00:17.0/ata1/host0/target0:0:0/0:0:0:0/block/sda (block)
KERNEL[760.665964] change   /devices/pci0000:00/0000:00:17.0/ata1/host0/target0:0:0/0:0:0:0/block/sda (block)
KERNEL[760.667413] change   /devices/pci0000:00/0000:00:17.0/ata1/host0/target0:0:0/0:0:0:0/block/sda/sda1 (block)
UDEV  [760.668116] change   /devices/pci0000:00/0000:00:17.0/ata1/host0/target0:0:0/0:0:0:0/block/sda/sda1 (block)
UDEV  [760.683925] change   /devices/pci0000:00/0000:00:17.0/ata1/host0/target0:0:0/0:0:0:0/block/sda (block)
UDEV  [760.690878] change   /devices/pci0000:00/0000:00:17.0/ata1/host0/target0:0:0/0:0:0:0/block/sda/sda1 (block)
KERNEL[760.697150] change   /devices/pci0000:00/0000:00:17.0/ata1/host0/target0:0:0/0:0:0:0/block/sda (block)
KERNEL[760.698944] change   /devices/pci0000:00/0000:00:17.0/ata1/host0/target0:0:0/0:0:0:0/block/sda/sda1 (block)

Running

sudo systemctl stop systemd-udevd systemd-udevd-kernel.socket systemd-udevd-control.socket    
sudo systemctl start systemd-udevd systemd-udevd-kernel.socket systemd-udevd-control.socket

temporarily fixes the issue until the next boot.

Here is my output from inxi --full --admin --filter --width

System:
  Kernel: 6.5.5-1-MANJARO arch: x86_64 bits: 64 compiler: gcc v: 13.2.1
    clocksource: tsc available: acpi_pm
    parameters: BOOT_IMAGE=/boot/vmlinuz-6.5-x86_64
    root=UUID=f54f97ce-0481-4e33-991c-8c70989bbb74 rw quiet splash apparmor=1
    security=apparmor udev.log_priority=3
  Desktop: GNOME v: 44.5 tk: GTK v: 3.24.38 wm: gnome-shell dm: GDM v: 44.1
    Distro: Manjaro Linux base: Arch Linux
Machine:
  Type: Laptop System: ASUSTeK product: UX410UAK v: 1.0
    serial: <superuser required>
  Mobo: ASUSTeK model: UX410UAK v: 1.0 serial: <superuser required>
    UEFI: American Megatrends v: UX410UAK.308 date: 11/08/2017
Battery:
  ID-1: BAT0 charge: 13.1 Wh (40.4%) condition: 32.4/48.3 Wh (67.0%)
    power: 15.2 W volts: 11.4 min: 11.4 model: ASUSTeK ASUS Battery type: Li-ion
    serial: N/A status: charging cycles: 1075
CPU:
  Info: model: Intel Core i5-7200U bits: 64 type: MT MCP arch: Amber/Kaby Lake
    note: check gen: core 7 level: v3 note: check built: 2017 process: Intel 14nm
    family: 6 model-id: 0x8E (142) stepping: 9 microcode: 0xF4
  Topology: cpus: 1x cores: 2 tpc: 2 threads: 4 smt: enabled cache:
    L1: 128 KiB desc: d-2x32 KiB; i-2x32 KiB L2: 512 KiB desc: 2x256 KiB
    L3: 3 MiB desc: 1x3 MiB
  Speed (MHz): avg: 3100 min/max: 400/3100 scaling: driver: intel_pstate
    governor: powersave cores: 1: 3100 2: 3100 3: 3100 4: 3100 bogomips: 21607
  Flags: avx avx2 ht lm nx pae sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx
  Vulnerabilities:
  Type: gather_data_sampling mitigation: Microcode
  Type: itlb_multihit status: KVM: VMX disabled
  Type: l1tf mitigation: PTE Inversion; VMX: conditional cache flushes, SMT
    vulnerable
  Type: mds mitigation: Clear CPU buffers; SMT vulnerable
  Type: meltdown mitigation: PTI
  Type: mmio_stale_data mitigation: Clear CPU buffers; SMT vulnerable
  Type: retbleed mitigation: IBRS
  Type: spec_rstack_overflow status: Not affected
  Type: spec_store_bypass mitigation: Speculative Store Bypass disabled via
    prctl
  Type: spectre_v1 mitigation: usercopy/swapgs barriers and __user pointer
    sanitization
  Type: spectre_v2 mitigation: IBRS, IBPB: conditional, STIBP: conditional,
    RSB filling, PBRSB-eIBRS: Not affected
  Type: srbds mitigation: Microcode
  Type: tsx_async_abort status: Not affected
Graphics:
  Device-1: Intel HD Graphics 620 vendor: ASUSTeK driver: i915 v: kernel
    arch: Gen-9.5 process: Intel 14nm built: 2016-20 ports: active: eDP-1
    empty: DP-1,HDMI-A-1,HDMI-A-2 bus-ID: 00:02.0 chip-ID: 8086:5916
    class-ID: 0300
  Device-2: Chicony USB2.0 HD UVC WebCam driver: uvcvideo type: USB rev: 2.0
    speed: 480 Mb/s lanes: 1 mode: 2.0 bus-ID: 1-6:2 chip-ID: 04f2:b57a
    class-ID: 0e02 serial: <filter>
  Display: wayland server: X.org v: 1.21.1.8 with: Xwayland v: 23.2.1
    compositor: gnome-shell driver: X: loaded: modesetting alternate: fbdev,vesa
    dri: iris gpu: i915 display-ID: 0
  Monitor-1: eDP-1 model: ChiMei InnoLux 0x14d2 built: 2016 res: 1920x1080
    dpi: 158 gamma: 1.2 size: 309x173mm (12.17x6.81") diag: 354mm (13.9")
    ratio: 16:9 modes: 1920x1080
  API: EGL v: 1.5 hw: drv: intel iris platforms: device: 0 drv: iris
    device: 1 drv: swrast surfaceless: drv: iris wayland: drv: iris x11:
    drv: iris inactive: gbm
  API: OpenGL v: 4.6 compat-v: 4.5 vendor: intel mesa v: 23.1.9-manjaro1.1
    glx-v: 1.4 direct-render: yes renderer: Mesa Intel HD Graphics 620 (KBL GT2)
    device-ID: 8086:5916 memory: 7.47 GiB unified: yes display-ID: :0.0
  API: Vulkan v: 1.3.264 layers: 4 device: 0 type: integrated-gpu name: Intel
    HD Graphics 620 (KBL GT2) driver: mesa intel v: 23.1.9-manjaro1.1
    device-ID: 8086:5916 surfaces: xcb,xlib,wayland
Audio:
  Device-1: Intel Sunrise Point-LP HD Audio vendor: ASUSTeK
    driver: snd_hda_intel v: kernel alternate: snd_soc_skl,snd_soc_avs
    bus-ID: 00:1f.3 chip-ID: 8086:9d71 class-ID: 0403
  API: ALSA v: k6.5.5-1-MANJARO status: kernel-api with: aoss
    type: oss-emulator tools: alsactl,alsamixer,amixer
  Server-1: sndiod v: N/A status: off tools: aucat,midicat,sndioctl
  Server-2: JACK v: 1.9.22 status: off
    tools: cadence,jack_control,jack_mixer,qjackctl
  Server-3: PipeWire v: 0.3.80 status: off with: wireplumber status: active
    tools: pw-cli,wpctl
  Server-4: PulseAudio v: 16.1 status: active with: 1: pulseaudio-alsa
    type: plugin 2: pulseaudio-jack type: module tools: pacat,pactl,pavucontrol
Network:
  Device-1: Intel Wireless 8260 driver: iwlwifi v: kernel pcie: gen: 1
    speed: 2.5 GT/s lanes: 1 bus-ID: 02:00.0 chip-ID: 8086:24f3 class-ID: 0280
  IF: wlp2s0 state: up mac: <filter>
  IF-ID-1: br-1ac68488355f state: down mac: <filter>
  IF-ID-2: br-676d578d147a state: down mac: <filter>
  IF-ID-3: br-8e3a90325e51 state: down mac: <filter>
  IF-ID-4: br-aec406e2658f state: down mac: <filter>
  IF-ID-5: br-d095adcc41e0 state: down mac: <filter>
  IF-ID-6: br-f46e94679df1 state: down mac: <filter>
  IF-ID-7: docker0 state: up speed: 10000 Mbps duplex: unknown mac: <filter>
  IF-ID-8: veth8a87b91 state: up speed: 10000 Mbps duplex: full mac: <filter>
Bluetooth:
  Device-1: Intel Bluetooth wireless interface driver: btusb v: 0.8 type: USB
    rev: 2.0 speed: 12 Mb/s lanes: 1 mode: 1.1 bus-ID: 1-8:3 chip-ID: 8087:0a2b
    class-ID: e001
  Report: btmgmt ID: hci0 rfk-id: 1 state: up address: <filter> bt-v: 4.2
    lmp-v: 8 status: discoverable: no pairing: no class-ID: 7c010c
Drives:
  Local Storage: total: 2.73 TiB used: 1.6 TiB (58.8%)
  SMART Message: Required tool smartctl not installed. Check --recommends
  ID-1: /dev/nvme0n1 maj-min: 259:0 vendor: Kingston model: SA2000M81000G
    size: 931.51 GiB block-size: physical: 512 B logical: 512 B speed: 31.6 Gb/s
    lanes: 4 tech: SSD serial: <filter> fw-rev: S5Z42105 temp: 34.9 C
    scheme: GPT
  ID-2: /dev/sda maj-min: 8:0 vendor: SanDisk model: SDSSDH3 2T00
    size: 1.82 TiB block-size: physical: 512 B logical: 512 B speed: 6.0 Gb/s
    tech: SSD serial: <filter> fw-rev: 20RL
Partition:
  ID-1: / raw-size: 784.92 GiB size: 771.52 GiB (98.29%)
    used: 626.48 GiB (81.2%) fs: ext4 dev: /dev/nvme0n1p5 maj-min: 259:5
  ID-2: /boot/efi raw-size: 100 MiB size: 96 MiB (96.00%)
    used: 29.6 MiB (30.9%) fs: vfat dev: /dev/nvme0n1p1 maj-min: 259:1
Swap:
  Kernel: swappiness: 60 (default) cache-pressure: 100 (default) zswap: yes
    compressor: zstd max-pool: 20%
  ID-1: swap-1 type: file size: 9.77 GiB used: 983.8 MiB (9.8%) priority: -2
    file: /swapfile
Sensors:
  System Temperatures: cpu: 77.0 C pch: 58.5 C mobo: N/A
  Fan Speeds (rpm): cpu: 3900
Info:
  Processes: 290 Uptime: 9m wakeups: 1 Memory: total: 8 GiB available: 7.65 GiB
  used: 6.01 GiB (78.6%) Init: systemd v: 254 default: graphical
  tool: systemctl Compilers: gcc: 13.2.1 clang: 16.0.6 Packages: 2808
  pm: nix-default pkgs: 0 pm: nix-sys pkgs: 0 pm: nix-usr pkgs: 0 pm: pacman
  pkgs: 2775 libs: 573 tools: gnome-software,pamac,yay pm: flatpak pkgs: 12
  pm: snap pkgs: 21 Shell: Zsh v: 5.9 running-in: gnome-terminal inxi: 3.3.30

What I tried:

  • Switching kernel versions
  • Updating whole system
  • Removing / Adding USB accessories

All help is greatly appreciated!

2

Udev watches /sys for changes. When ever you unplug or plugin a devices (thus there was a change recognized), udev will trigger the udev rules again.

Please exclude a loose contact on that SSD connected via SATA. Heat, dust or a faulty cable/connector can also result of that.

1 Like
3

Hey megavolt, thank you for your quick answer. I will open the notebook when I get the necessary screwdriver. However, it seems very unlikely to me because udevadm logs these changes in quick succession before restarting the services and upon restarting I get no logs for a few hours until it starts again.

4

At least I can only say that there must be some hardware or firmware or module related problem. Udev just watches /sys and apply rules on any change. And if the kernel report many changes, then udev will also apply as many times. So in my perspective udev is not the problem here.

1 Like
5

seems like i am experiencing the very same issue:

  • high udev CPU load (80% for me and often ending in a system freeze when trying to keep working)
  • the reported work-around of stopping and restarting the udev services helps me as well, until the next boot.

this started very recently and is probably related with updating to the 6.1.55 kernel

however, i also have 5.15.133 installed and i am not having any such issues with this kernel. so i am back to 5.15 for now…

6

Thanks for the pointer. I think I can probably rule out a hardware issue at this point because it constantly only happens after rebooting and has nothing to do with overheating. Which kernel modules would I have to look at judging from the udev log?

7

definitely no hardware problem in my case. everything still works fine with 5.15 and my problem with 6.1 only started earlier this month with the latest (kernel) updates.

i noticed that in my case the 2 drives/partitions affected (as in constantly being swapped back and forth) are part of my ‘fake’ raid1 (by way of dm_mod, dmraid, ntfs-3g). if i disable my mount script (systemd service) during boot and run it manually after boot everything works fine in 6.1 too, without udev hickups.

8

Unfortunately, switchin to 5.15 did not help me with my problem. I don’t even know where to look from here, where did you find your mount scripts? I also have 2 drives, one even with NTFS partitions.

9

bummer, was hoping the 5.15 branch was not affected. strange that it works for me though. can you avoid the problem (like i can) if you don’t mount your drives during boot (fstab) but later, manually?

so our systems have in common 2 hard disks, which is probably not what most developers have these days. NTFS may or may not contribute to the problem.

i am no expert either but this all looks like a race condition to me. i see from other threads that there has been some recent kernel work on ATA drives with the latest kernels causing problems for hard disks during shut down. our problem is a different but started with an earlier kernel version, so i suspect hard disk regressions came in two stages. hopefully, the kernel fix for the shutdown issue will fix our problem too.

regarding my NTFS RAID1: i was loosely following instructions from https://wiki.archlinux.org/title/Install_Arch_Linux_with_Fake_RAID to get it working under linux. because an extra kernel module needs to be loaded i cannot mount with a simple fstab enty but had to create a systemd service and script to be executed during boot. you probably just have 2 fstab entries for your hard disks, so it is not my ‘fake’ raid and mounting method that is causing the problem. some sort of relief at least.

10

How are you guys mounting the NTFS drive? On kernel 6.5.5-1 I have no problems with

$ lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda           8:0    0 476.9G  0 disk 
├─sda1        8:1    0   100M  0 part 
└─sda2        8:2    0 476.8G  0 part 
sdb           8:16   0   1.8T  0 disk 
└─sdb1        8:17   0   1.8T  0 part 
nvme0n1     259:0    0 931.5G  0 disk 
├─nvme0n1p1 259:1    0   300M  0 part /boot/efi
└─nvme0n1p2 259:2    0 931.2G  0 part /

where sda is an old Windows 7 NTFS ssd and sdb is an even older Windows NTFS hdd. I don’t mount either of them in /etc/fstab though, because I don’t use them very often at the moment;

$ cat /etc/fstab
# <file system>             <mount point>  <type>  <options>  <dump>  <pass>
UUID=1E46-8640                            /boot/efi      vfat    umask=0077 0 2
UUID=fb91cf1b-750b-4956-a8a8-d2d211a5dcf8 /              ext4    defaults,noatime 0 1
11

my ‘fake’ raid1 (legacy from win7):

sdc                jmicron_raid_member 0.1                                                                       
├─sdc1             ntfs                      Data            A07299E37299BF0A                                    
└─jmicron_DATA                                                                                                   
  └─jmicron_DATAp1 ntfs                      Data            A07299E37299BF0A                       58.6G    97% /mnt/Data
sdd                jmicron_raid_member 0.1                                                                       
├─sdd1             ntfs                      Data            A07299E37299BF0A                                    
└─jmicron_DATA                                                                                                   
  └─jmicron_DATAp1 ntfs                      Data            A07299E37299BF0A                       58.6G    97% /mnt/Data

is mounted with this this script:

$ cat /usr/local/sbin/jmicron-raid 
#!/bin/bash

##Bash script to activate and mount jmicron "fakeraid"

modprobe dm_mod
dmraid -ay jmicron_DATA
ntfs-3g -o windows_names,recover,big_writes /dev/mapper/jmicron_DATAp1 /mnt/Data

i have no problems at all (incl kernel 6.1.55) when i boot up first and run this script manually. however, i used to have a systemd service connecting it during boot:

$ cat /etc/systemd/system/jmicron-raid.service 
[Unit]
Description=Set up JMicron Raid

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/jmicron-raid
RemainAfterExit=true

[Install]
WantedBy=multi-user.target

this worked fine for me for the last couple of years and still does for 5.15.133, my fallback kernel. however, since kernel 6.1.55 i experience the high load ‘race condition’-like fast back & forth switching/swapping of the the drives if i have my mount service active:

UDEV  [78.979637] change   /devices/pci0000:00/0000:00:1c.3/0000:04:00.0/ata9/host8/target8:0:0/8:0:0:0/block/sdc/sdc1 (block)
UDEV  [78.993077] change   /devices/pci0000:00/0000:00:1c.3/0000:04:00.0/ata10/host9/target9:0:0/9:0:0:0/block/sdd/sdd1 (block)
UDEV  [78.994486] change   /devices/pci0000:00/0000:00:1c.3/0000:04:00.0/ata9/host8/target8:0:0/8:0:0:0/block/sdc (block)
KERNEL[78.999288] change   /devices/pci0000:00/0000:00:1c.3/0000:04:00.0/ata9/host8/target8:0:0/8:0:0:0/block/sdc (block)
KERNEL[79.000361] change   /devices/pci0000:00/0000:00:1c.3/0000:04:00.0/ata9/host8/target8:0:0/8:0:0:0/block/sdc/sdc1 (block)
UDEV  [79.008973] change   /devices/pci0000:00/0000:00:1c.3/0000:04:00.0/ata10/host9/target9:0:0/9:0:0:0/block/sdd (block)
KERNEL[79.013872] change   /devices/pci0000:00/0000:00:1c.3/0000:04:00.0/ata10/host9/target9:0:0/9:0:0:0/block/sdd (block)
KERNEL[79.015165] change   /devices/pci0000:00/0000:00:1c.3/0000:04:00.0/ata10/host9/target9:0:0/9:0:0:0/block/sdd/sdd1 (block)
UDEV  [79.015718] change   /devices/pci0000:00/0000:00:1c.3/0000:04:00.0/ata9/host8/target8:0:0/8:0:0:0/block/sdc/sdc1 (block)
UDEV  [79.026606] change   /devices/pci0000:00/0000:00:1c.3/0000:04:00.0/ata10/host9/target9:0:0/9:0:0:0/block/sdd/sdd1 (block)
UDEV  [79.027538] change   /devices/pci0000:00/0000:00:1c.3/0000:04:00.0/ata9/host8/target8:0:0/8:0:0:0/block/sdc (block)
KERNEL[79.034683] change   /devices/pci0000:00/0000:00:1c.3/0000:04:00.0/ata9/host8/target8:0:0/8:0:0:0/block/sdc (block)
KERNEL[79.035709] change   /devices/pci0000:00/0000:00:1c.3/0000:04:00.0/ata9/host8/target8:0:0/8:0:0:0/block/sdc/sdc1 (block)
UDEV  [79.042699] change   /devices/pci0000:00/0000:00:1c.3/0000:04:00.0/ata10/host9/target9:0:0/9:0:0:0/block/sdd (block)
KERNEL[79.048397] change   /devices/pci0000:00/0000:00:1c.3/0000:04:00.0/ata10/host9/target9:0:0/9:0:0:0/block/sdd (block)
KERNEL[79.049500] change   /devices/pci0000:00/0000:00:1c.3/0000:04:00.0/ata10/host9/target9:0:0/9:0:0:0/block/sdd/sdd1 (block)
UDEV  [79.050194] change   /devices/pci0000:00/0000:00:1c.3/0000:04:00.0/ata9/host8/target8:0:0/8:0:0:0/block/sdc/sdc1 (block)
UDEV  [79.061679] change   /devices/pci0000:00/0000:00:1c.3/0000:04:00.0/ata9/host8/target8:0:0/8:0:0:0/block/sdc (block)
UDEV  [79.066933] change   /devices/pci0000:00/0000:00:1c.3/0000:04:00.0/ata10/host9/target9:0:0/9:0:0:0/block/sdd/sdd1 (block)
KERNEL[79.069407] change   /devices/pci0000:00/0000:00:1c.3/0000:04:00.0/ata9/host8/target8:0:0/8:0:0:0/block/sdc (block)
KERNEL[79.070590] change   /devices/pci0000:00/0000:00:1c.3/0000:04:00.0/ata9/host8/target8:0:0/8:0:0:0/block/sdc/sdc1 (block)
UDEV  [79.076430] change   /devices/pci0000:00/0000:00:1c.3/0000:04:00.0/ata10/host9/target9:0:0/9:0:0:0/block/sdd (block)
UDEV  [79.080762] change   /devices/pci0000:00/0000:00:1c.3/0000:04:00.0/ata9/host8/target8:0:0/8:0:0:0/block/sdc/sdc1 (block)

to be closer to Fliwatt’s and my set-up you should test what happens if you try to mount your 2 hard disks during boot, i.e. by putting them in fstab. i am quite curious about the outcome.

12

Thanks for all the investigation. I can confirm that on my setup unticking the checkbox “Mount at System startup” in gnome-disk-utility solves the problem.

image
image950×613 65.6 KB

I don’t know how exactly this works because my /etc/fstab seems to stay untouched whenever I change this setting:

# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a device; this may
# be used with UUID= as a more robust way to name devices that works even if
# disks are added and removed. See fstab(5).
#
# <file system>             <mount point>  <type>  <options>  <dump>  <pass>
UUID=12F9-AA0A                            /boot/efi      vfat    umask=0077 0 2
UUID=f54f97ce-0481-4e33-991c-8c70989bbb74 /              ext4    defaults,noatime 0 1
/swapfile none swap defaults 0 0
/dev/disk/by-uuid/6ED64665D6462E21 /mnt/6ED64665D6462E21 auto nosuid,nodev,nofail,x-gvfs-show 0 0

and the utitlity is not asking for sudo privileges but it works!

After rebooting and putting the checkmark in again, my output of lsblk is this (i left out the snaps):

sda           8:0    0   1,8T  0 disk 
└─sda1        8:1    0   1,8T  0 part /mnt/6ED64665D6462E21
nvme0n1     259:0    0 931,5G  0 disk 
├─nvme0n1p1 259:1    0   100M  0 part /boot/efi
├─nvme0n1p2 259:2    0    16M  0 part 
├─nvme0n1p3 259:3    0   146G  0 part 
├─nvme0n1p4 259:4    0   515M  0 part 
└─nvme0n1p5 259:5    0 784,9G  0 part /var/lib/snapd/snap

Do you guys think that downgrading systemd might help? This bug started occuring approximately at the beginning of October which coincides with the release of systemd 254.5 (though the changelog does not give me much information). Sorry, I am relatively new to the Linux world so I am not sure if downgrading systemd is a good idea and I also don’t want to break my production system.

13

just upgraded to the latest stable manjaro update 2023-11-06, which updated my 6.1 kernel to 6.1.60. this miraculously resolved my issues regarding high CPU load / udev. the system boots and mounts all (physical/NTFS) hard drives alright again.

14

This topic was automatically closed 3 hours after the last reply. New replies are no longer allowed.

15

Reopened at request of OP

16

@captaindet Unfortunately, that did not do the trick for me, I am still having the exact same issue.

EDIT: Suddenly, with the latest Manjaro update and the Kernel update to 6.5.11-1 it started to work on this kernel!

17

This topic was automatically closed 3 hours after the last reply. New replies are no longer allowed.