Hello, since some time (a few weeks I think, I cannot really trace it back to the moment it started to happen) systemd is not able (almost all the times) to properly kill a job during shutdown which results in an annoying 2 min timeout. I know I can change the timeout, but I would like to get to the bottom of this problem and solve is rather than using a workaround.
I have already followed the Debugging instructions at the systemd website, and the (relevant lines of the) shutdown log are the following:
[ 1707.861271] systemd[1]: user@1000.service: State 'stop-sigterm' timed out. Killing.
[ 1707.861358] systemd[1]: user@1000.service: Killing process 1388 (systemd) with signal SIGKILL.
[ 1707.862109] systemd[1]: user@1000.service: Killing process 1764 (python3) with signal SIGKILL.
[ 1707.862196] systemd[1]: user@1000.service: Killing process 1971 (libinput-debug-) with signal SIGKILL.
[ 1707.862822] systemd[1]: user@1000.service: Killing process 2007 (pipewire) with signal SIGKILL.
[ 1707.862914] systemd[1]: user@1000.service: Killing process 2009 (pipewire-media-) with signal SIGKILL.
[ 1707.863292] systemd[1]: user@1000.service: Killing process 7433 (dbus-daemon) with signal SIGKILL.
[ 1707.864548] systemd[1]: user@1000.service: Changed stop-sigterm -> stop-sigkill
[ 1707.865355] systemd[1]: systemd-journald.service: Received EPOLLHUP on stored fd 212 (stored), closing.
[ 1707.865503] systemd[1]: systemd-journald.service: Received EPOLLHUP on stored fd 204 (stored), closing.
[ 1707.865595] systemd[1]: systemd-journald.service: Received EPOLLHUP on stored fd 203 (stored), closing.
[ 1707.865684] systemd[1]: Received SIGCHLD from PID 2009 (pipewire-media-).
[ 1707.865704] systemd[1]: Child 2009 (pipewire-media-) died (code=killed, status=9/KILL)
[ 1707.865760] systemd[1]: user@1000.service: Failed to read oom_kill field of memory.events cgroup attribute: No such file or directory
[ 1707.865762] systemd[1]: user@1000.service: Child 2009 belongs to user@1000.service.
[ 1707.865998] systemd[1]: systemd-journald.service: Received EPOLLHUP on stored fd 146 (stored), closing.
[ 1707.872096] systemd[1]: systemd-journald.service: Received EPOLLHUP on stored fd 12 (stored), closing.
[ 1707.878827] systemd[1]: Received SIGCHLD from PID 1389 ((sd-pam)).
[ 1707.878861] systemd[1]: Child 1388 (systemd) died (code=killed, status=9/KILL)
[ 1707.878922] systemd[1]: user@1000.service: Failed to read oom_kill field of memory.events cgroup attribute: No such file or directory
[ 1707.878925] systemd[1]: user@1000.service: Child 1388 belongs to user@1000.service.
[ 1707.878933] systemd[1]: user@1000.service: Main process exited, code=killed, status=9/KILL
[ 1707.879068] systemd[1]: user@1000.service: Killing process 1971 (libinput-debug-) with signal SIGKILL.
[ 1707.879279] systemd[1]: user@1000.service: Failed with result 'timeout'.
[ 1707.879349] systemd[1]: user@1000.service: Unit process 1971 (libinput-debug-) remains running after unit stopped.
[ 1707.879476] systemd[1]: user@1000.service: Service restart not allowed.
[ 1707.879484] systemd[1]: user@1000.service: Changed stop-sigkill -> failed
[ 1707.880036] systemd[1]: user@1000.service: Failed to destroy cgroup /user.slice/user-1000.slice/user@1000.service, ignoring: Device or resource busy
[ 1707.880066] systemd[1]: user@1000.service: Job 2293 user@1000.service/stop finished, result=done
[ 1707.880070] systemd[1]: Stopped User Manager for UID 1000.
[ 1707.880347] systemd[1]: user@1000.service: Unit entered failed state.
[ 1707.880391] kauditd_printk_skb: 8 callbacks suppressed
[ 1707.880393] audit: type=1131 audit(1610212868.760:165): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=user@1000 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
[ 1707.880466] systemd[1]: Child 1389 ((sd-pam)) died (code=killed, status=9/KILL)
[ 1707.880531] systemd[1]: user@1000.service: Failed to read oom_kill field of memory.events cgroup attribute: No such file or directory
[ 1707.880534] systemd[1]: user@1000.service: Child 1389 belongs to user@1000.service.
[ 1707.880563] systemd[1]: Child 2007 (pipewire) died (code=killed, status=9/KILL)
[ 1707.880603] systemd[1]: user@1000.service: Failed to read oom_kill field of memory.events cgroup attribute: No such file or directory
[ 1707.880605] systemd[1]: user@1000.service: Child 2007 belongs to user@1000.service.
[ 1707.880638] systemd[1]: Child 1764 (python3) died (code=killed, status=9/KILL)
[ 1707.880678] systemd[1]: user@1000.service: Failed to read oom_kill field of memory.events cgroup attribute: No such file or directory
[ 1707.880681] systemd[1]: user@1000.service: Child 1764 belongs to user@1000.service.
[ 1707.880720] systemd[1]: Child 7433 (dbus-daemon) died (code=killed, status=9/KILL)
[ 1707.880757] systemd[1]: user@1000.service: Failed to read oom_kill field of memory.events cgroup attribute: No such file or directory
[ 1707.880760] systemd[1]: user@1000.service: Child 7433 belongs to user@1000.service.
[ 1707.880784] systemd[1]: user-1000.slice: stopping held back, waiting for: user-runtime-dir@1000.service
[ 1707.880788] systemd[1]: basic.target: stopping held back, waiting for: dbus.service
[ 1707.880792] systemd[1]: shutdown.target: starting held back, waiting for: swap.target
[ 1707.880800] systemd[1]: sysinit.target: stopping held back, waiting for: basic.target
[ 1707.880804] systemd[1]: dbus.service: stopping held back, waiting for: wpa_supplicant.service
[ 1707.880808] systemd[1]: systemd-user-sessions.service: stopping held back, waiting for: user-1000.slice
[ 1707.881124] systemd[1]: Failed to read pids.max attribute of cgroup root, ignoring: No data available
[ 1707.881162] systemd[1]: user-runtime-dir@1000.service: About to execute /usr/lib/systemd/systemd-user-runtime-dir stop 1000
[ 1707.881402] systemd[1]: user-runtime-dir@1000.service: Forked /usr/lib/systemd/systemd-user-runtime-dir as 7436
[ 1707.881528] systemd[1]: user-runtime-dir@1000.service: Changed exited -> stop
[ 1707.881542] systemd[1]: Stopping User Runtime Directory /run/user/1000...
[ 1707.882153] systemd[7436]: user-runtime-dir@1000.service: Executing: /usr/lib/systemd/systemd-user-runtime-dir stop 1000
[ 1707.883219] systemd-journald[195]: Successfully sent stream file descriptor to service manager.
[ 1707.883292] systemd[1]: systemd-journald.service: Got notification message from PID 195 (FDSTORE=1)
[ 1707.883306] systemd[1]: systemd-journald.service: Added fd 12 (n/a) to fd store.
[ 1707.885025] systemd-user-runtime-dir[7436]: Will remove /run/user/1000
[ 1707.886419] systemd[1]: run-user-1000.mount: Succeeded.
[ 1707.886444] systemd[1]: run-user-1000.mount: Changed mounted -> dead
[ 1707.886519] systemd[1]: run-user-1000.mount: Job 2423 run-user-1000.mount/stop finished, result=done
[ 1707.886522] systemd[1]: Unmounted /run/user/1000.
[ 1707.886636] systemd[1]: run-user-1000.mount: Collecting.
[ 1707.886744] systemd[1]: Received SIGCHLD from PID 7436 (systemd-user-ru).
[ 1707.886763] systemd[1]: Child 7436 (systemd-user-ru) died (code=exited, status=0/SUCCESS)
[ 1707.886809] systemd[1]: user-runtime-dir@1000.service: Failed to read oom_kill field of memory.events cgroup attribute: No such file or directory
[ 1707.886812] systemd[1]: user-runtime-dir@1000.service: Child 7436 belongs to user-runtime-dir@1000.service.
[ 1707.886817] systemd[1]: user-runtime-dir@1000.service: Control process exited, code=exited, status=0/SUCCESS
[ 1707.886819] systemd[1]: user-runtime-dir@1000.service: Got final SIGCHLD for state stop.
[ 1707.886910] systemd[1]: user-runtime-dir@1000.service: Succeeded.
[ 1707.886926] systemd[1]: user-runtime-dir@1000.service: Service restart not allowed.
[ 1707.886930] systemd[1]: user-runtime-dir@1000.service: Changed stop -> dead
[ 1707.887205] systemd[1]: user-runtime-dir@1000.service: Job 2294 user-runtime-dir@1000.service/stop finished, result=done
[ 1707.887208] systemd[1]: Stopped User Runtime Directory /run/user/1000.
[ 1707.887319] audit: type=1131 audit(1610212868.766:166): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=user-runtime-dir@1000 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 1707.887362] systemd[1]: systemd-journald.service: Received EPOLLHUP on stored fd 12 (stored), closing.
[ 1707.887382] systemd[1]: user-1000.slice changed active -> dead
[ 1707.887543] systemd[1]: user-1000.slice: Failed to destroy cgroup /user.slice/user-1000.slice, ignoring: Device or resource busy
[ 1707.887561] systemd[1]: user-1000.slice: Job 2291 user-1000.slice/stop finished, result=done
[ 1707.887564] systemd[1]: Removed slice User Slice of UID 1000.
[ 1707.887649] systemd[1]: user.slice: stopping held back, waiting for: slices.target
[ 1707.887654] systemd[1]: basic.target: stopping held back, waiting for: dbus.service
[ 1707.887657] systemd[1]: local-fs-pre.target: stopping held back, waiting for: local-fs.target
[ 1707.887660] systemd[1]: local-fs.target: stopping held back, waiting for: sysinit.target
[ 1707.887663] systemd[1]: shutdown.target: starting held back, waiting for: swap.target
[ 1707.887668] systemd[1]: swap.target: stopping held back, waiting for: tmp.mount
[ 1707.887675] systemd[1]: sysinit.target: stopping held back, waiting for: basic.target
[ 1707.887678] systemd[1]: umount.target: starting held back, waiting for: dev-disk-by\x2dpath-pci\x2d0000:3b:00.0\x2dnvme\x2d1\x2dpart1.swap
[ 1707.887682] systemd[1]: dbus.service: stopping held back, waiting for: wpa_supplicant.service
[ 1707.887974] systemd[1]: Failed to read pids.max attribute of cgroup root, ignoring: No data available
[ 1707.888007] systemd[1]: systemd-user-sessions.service: About to execute /usr/lib/systemd/systemd-user-sessions stop
[ 1707.888216] systemd[1]: systemd-user-sessions.service: Forked /usr/lib/systemd/systemd-user-sessions as 7437
[ 1707.888312] systemd[1]: systemd-user-sessions.service: Changed exited -> stop
[ 1707.888323] systemd[1]: Stopping Permit User Sessions...
[ 1707.888892] systemd[7437]: systemd-user-sessions.service: Executing: /usr/lib/systemd/systemd-user-sessions stop
[ 1707.889813] systemd-journald[195]: Successfully sent stream file descriptor to service manager.
[ 1707.889894] systemd[1]: systemd-journald.service: Got notification message from PID 195 (FDSTORE=1)
[ 1707.889910] systemd[1]: systemd-journald.service: Added fd 12 (n/a) to fd store.
[ 1707.892328] systemd[1]: Received SIGCHLD from PID 7437 (systemd-user-se).
[ 1707.892358] systemd[1]: Child 7437 (systemd-user-se) died (code=exited, status=0/SUCCESS)
[ 1707.892411] systemd[1]: systemd-user-sessions.service: Failed to read oom_kill field of memory.events cgroup attribute: No such file or directory
[ 1707.892414] systemd[1]: systemd-user-sessions.service: Child 7437 belongs to systemd-user-sessions.service.
[ 1707.892420] systemd[1]: systemd-user-sessions.service: Control process exited, code=exited, status=0/SUCCESS
[ 1707.892423] systemd[1]: systemd-user-sessions.service: Got final SIGCHLD for state stop.
[ 1707.892516] systemd[1]: systemd-user-sessions.service: Succeeded.
[ 1707.892532] systemd[1]: systemd-user-sessions.service: Service restart not allowed.
[ 1707.892538] systemd[1]: systemd-user-sessions.service: Changed stop -> dead
[ 1707.892847] systemd[1]: systemd-user-sessions.service: Job 2325 systemd-user-sessions.service/stop finished, result=done
[ 1707.892851] systemd[1]: Stopped Permit User Sessions.
[ 1707.893111] audit: type=1131 audit(1610212868.773:167): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-user-sessions comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 1707.893129] systemd[1]: systemd-journald.service: Received EPOLLHUP on stored fd 12 (stored), closing.
[ 1707.893152] systemd[1]: basic.target: stopping held back, waiting for: dbus.service
[ 1707.893161] systemd[1]: network.target changed active -> dead
[ 1707.893165] systemd[1]: network.target: Job 2403 network.target/stop finished, result=done
[ 1707.893169] systemd[1]: Stopped target Network.
[ 1707.893294] systemd[1]: nss-user-lookup.target changed active -> dead
[ 1707.893300] systemd[1]: nss-user-lookup.target: Job 2384 nss-user-lookup.target/stop finished, result=done
[ 1707.893303] systemd[1]: Stopped target User and Group Name Lookups.
[ 1707.893432] systemd[1]: remote-fs.target changed active -> dead
[ 1707.893435] systemd[1]: remote-fs.target: Job 2349 remote-fs.target/stop finished, result=done
[ 1707.893438] systemd[1]: Stopped target Remote File Systems.
[ 1707.893556] systemd[1]: shutdown.target: starting held back, waiting for: swap.target
[ 1707.893568] systemd[1]: sysinit.target: stopping held back, waiting for: basic.target
[ 1707.893574] systemd[1]: home.mount: stopping held back, waiting for: local-fs.target
[ 1707.893590] systemd[1]: NetworkManager.service: Changed running -> stop-sigterm
[ 1707.893595] systemd[1]: Stopping Network Manager...
[ 1707.893738] systemd[1]: wpa_supplicant.service: Changed running -> stop-sigterm
[ 1707.893745] systemd[1]: Stopping WPA supplicant...
[ 1707.895578] systemd[1]: Received SIGCHLD from PID 1971 (libinput-debug-).
[ 1707.895606] systemd[1]: Child 1971 (libinput-debug-) died (code=killed, status=9/KILL)
[ 1707.895677] systemd[1]: user@1000.service: Failed to read oom_kill field of memory.events cgroup attribute: No such file or directory
[ 1707.895682] systemd[1]: user@1000.service: Child 1971 belongs to user@1000.service.
[ 1707.895721] systemd[1]: user@1000.service: Control group is empty.
So, I believe the issue is with libinput-debug-event
, however I am stuck, I don’t know how to continue to solve the problem…
Here is my inxi
System:
Kernel: 5.10.2-2-MANJARO x86_64 bits: 64 compiler: gcc v: 10.2.0
parameters: BOOT_IMAGE=/boot/vmlinuz-5.10-x86_64
root=UUID=9eb44dd1-1630-4db8-8f51-9542e721051b rw pcie_aspm=force pci=noaer
nmi_watchdog=0 i915.modeset=1 i915.enable_fbc=1 i915.enable_psr=2
i915.enable_dc=2 i915.disable_power_well=1 systemd.log_level=debug
systemd.log_target=kmsg log_buf_len=1M printk.devkmsg=on enforcing=0
Desktop: GNOME 3.38.2 tk: GTK 3.24.24 wm: gnome-shell dm: GDM 3.38.2.1
Distro: Manjaro Linux
Machine:
Type: Laptop System: Dell product: XPS 13 9360 v: N/A serial: <filter>
Chassis: type: 9 serial: <filter>
Mobo: Dell model: 05JK94 v: A00 serial: <filter> UEFI: Dell v: 2.13.0
date: 11/14/2019
Battery:
ID-1: BAT0 charge: 60.0 Wh condition: 47.5/60.0 Wh (79%) volts: 8.5/7.6
model: SMP DELL TP1GT61 type: Li-poly serial: <filter> status: Full
CPU:
Info: Dual Core model: Intel Core i5-7200U bits: 64 type: MT MCP
arch: Amber/Kaby Lake note: check family: 6 model-id: 8E (142) stepping: 9
microcode: DE L2 cache: 3 MiB
flags: avx avx2 lm nx pae sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx
bogomips: 21607
Speed: 800 MHz min/max: 400/3100 MHz Core speeds (MHz): 1: 800 2: 800 3: 800
4: 800
Vulnerabilities: Type: itlb_multihit status: KVM: VMX disabled
Type: l1tf
mitigation: PTE Inversion; VMX: conditional cache flushes, SMT vulnerable
Type: mds mitigation: Clear CPU buffers; SMT vulnerable
Type: meltdown mitigation: PTI
Type: spec_store_bypass
mitigation: Speculative Store Bypass disabled via prctl and seccomp
Type: spectre_v1
mitigation: usercopy/swapgs barriers and __user pointer sanitization
Type: spectre_v2 mitigation: Full generic retpoline, IBPB: conditional,
IBRS_FW, STIBP: conditional, RSB filling
Type: srbds mitigation: Microcode
Type: tsx_async_abort status: Not affected
Graphics:
Device-1: Intel HD Graphics 620 vendor: Dell driver: i915 v: kernel
bus ID: 00:02.0 chip ID: 8086:5916
Device-2: Microdia Integrated Webcam HD type: USB driver: uvcvideo
bus ID: 1-5:3 chip ID: 0c45:670c
Display: wayland server: X.org 1.20.10 compositor: gnome-shell driver: intel
unloaded: modesetting alternate: fbdev,vesa display ID: 0
resolution: <missing: xdpyinfo>
OpenGL: renderer: Mesa Intel HD Graphics 620 (KBL GT2) v: 4.6 Mesa 20.3.1
direct render: Yes
Audio:
Device-1: Intel Sunrise Point-LP HD Audio vendor: Dell driver: snd_hda_intel
v: kernel alternate: snd_soc_skl bus ID: 00:1f.3 chip ID: 8086:9d71
Device-2: Logitech HD Webcam C525 type: USB driver: snd-usb-audio,uvcvideo
bus ID: 3-1.3:4 chip ID: 046d:0826 serial: <filter>
Sound Server: ALSA v: k5.10.2-2-MANJARO
Network:
Device-1: Qualcomm Atheros QCA6174 802.11ac Wireless Network Adapter
vendor: Bigfoot Networks driver: ath10k_pci v: kernel port: f040
bus ID: 3a:00.0 chip ID: 168c:003e
IF: wlp58s0 state: up mac: <filter>
Device-2: Qualcomm Atheros QCA61x4 Bluetooth 4.0 type: USB driver: btusb
bus ID: 1-3:2 chip ID: 0cf3:e300
Device-3: Realtek RTL8153 Gigabit Ethernet Adapter type: USB driver: r8152
bus ID: 4-1.2:3 chip ID: 0bda:8153 serial: <filter>
IF: enp57s0u1u2 state: down mac: <filter>
Drives:
Local Storage: total: 953.87 GiB used: 162.73 GiB (17.1%)
SMART Message: Unable to run smartctl. Root privileges required.
ID-1: /dev/nvme0n1 maj-min: 259:0 vendor: A-Data model: SX8200PNP
size: 953.87 GiB block size: physical: 512 B logical: 512 B speed: 31.6 Gb/s
lanes: 4 serial: <filter> rev: 42AZS6AC temp: 36.9 C
Partition:
ID-1: / raw size: 128 GiB size: 124.99 GiB (97.65%) used: 50.88 GiB (40.7%)
fs: ext4 dev: /dev/nvme0n1p2 maj-min: 259:2
ID-2: /boot/efi raw size: 512 MiB size: 511 MiB (99.80%)
used: 280 KiB (0.1%) fs: vfat dev: /dev/nvme0n1p4 maj-min: 259:4
ID-3: /home raw size: 817.36 GiB size: 803.53 GiB (98.31%)
used: 111.85 GiB (13.9%) fs: ext4 dev: /dev/nvme0n1p3 maj-min: 259:3
Swap:
Kernel: swappiness: 60 (default) cache pressure: 100 (default)
ID-1: swap-1 type: partition size: 8 GiB used: 0 KiB (0.0%) priority: -2
dev: /dev/nvme0n1p1 maj-min: 259:1
Sensors:
System Temperatures: cpu: 56.0 C mobo: N/A
Fan Speeds (RPM): cpu: 0
Info:
Processes: 204 Uptime: 9m wakeups: 1 Memory: 7.5 GiB used: 2.82 GiB (37.7%)
Init: systemd v: 247 Compilers: gcc: 10.2.0 clang: 11.0.0 Packages: 1572
pacman: 1552 lib: 429 flatpak: 20 Shell: Zsh v: 5.8 running in: tilix
inxi: 3.2.01
Cheers