Systemd: Freezing execution

Support
1

At first there seemed to be no problem but since yesterday I have systemd freezing. No clue how or where to debug this. Any help will be appreciated.

This is the message I get:

Broadcast message from systemd-journald@roland-xps (Sun 2021-07-25 22:32:01 CEST):

systemd[1]: Caught <ABRT>, dumped core as pid 35770.


Broadcast message from systemd-journald@roland-xps (Sun 2021-07-25 22:32:01 CEST):

systemd[1]: Freezing execution.

This is the journal log of the coredump

           PID: 35770 (systemd)
           UID: 0 (root)
           GID: 0 (root)
        Signal: 6 (ABRT)
     Timestamp: Sun 2021-07-25 22:32:01 CEST (5min ago)
  Command Line: /sbin/init
    Executable: /usr/lib/systemd/systemd
 Control Group: /init.scope
          Unit: init.scope
         Slice: -.slice
       Boot ID: d597d4d4236347f58dba94b7378ae371
    Machine ID: a7b3bc1365d44bf8a29140db98c319c9
      Hostname: roland-xps
       Storage: /var/lib/systemd/coredump/core.systemd.0.d597d4d4236347f58dba94b7378ae371.35770.1627245121000000.zst (present)
     Disk Size: 446.3K
       Message: Process 35770 (systemd) of user 0 dumped core.
                
                Stack trace of thread 35770:
                #0  0x00007f6b9d7e601b kill (libc.so.6 + 0x3d01b)
                #1  0x00005569c0df2e97 n/a (systemd + 0x3ce97)
                #2  0x00007f6b9d988870 __restore_rt (libpthread.so.0 + 0x13870)
                #3  0x00007f6b9d7e5d22 raise (libc.so.6 + 0x3cd22)
                #4  0x00007f6b9d7cf862 abort (libc.so.6 + 0x26862)
                #5  0x00007f6b9db2b972 n/a (libsystemd-shared-248.so + 0x62972)
                #6  0x00007f6b9dcca980 n/a (libsystemd-shared-248.so + 0x201980)
                #7  0x00007f6b9dcd303e n/a (libsystemd-shared-248.so + 0x20a03e)
                #8  0x00007f6b9dcd3a8e sd_event_prepare (libsystemd-shared-248.so + 0x20aa8e)
                #9  0x00007f6b9dcd6bb3 sd_event_run (libsystemd-shared-248.so + 0x20dbb3)
                #10 0x00005569c0e375e8 n/a (systemd + 0x815e8)
                #11 0x00005569c0defe2d n/a (systemd + 0x39e2d)
                #12 0x00007f6b9d7d0b25 __libc_start_main (libc.so.6 + 0x27b25)
                #13 0x00005569c0df2a0e n/a (systemd + 0x3ca0e)

The coredump file it produces is 20MB and not usable for displaying here.

This is the neofetch output:

██████████████████  ████████   rolandd@roland-xps 
██████████████████  ████████   ------------------ 
██████████████████  ████████   OS: Manjaro Linux x86_64 
██████████████████  ████████   Host: XPS 17 9700 
████████            ████████   Kernel: 5.13.4-1-MANJARO 
████████  ████████  ████████   Uptime: 1 hour, 7 mins 
████████  ████████  ████████   Packages: 1488 (pacman) 
████████  ████████  ████████   Shell: zsh 5.8 
████████  ████████  ████████   Resolution: 2560x1440 
████████  ████████  ████████   DE: Plasma 5.22.3 
████████  ████████  ████████   WM: KWin 
████████  ████████  ████████   Theme: Breeze Dark [Plasma], Default [GTK2/3] 
████████  ████████  ████████   Icons: [Plasma], breeze-dark [GTK2/3] 
████████  ████████  ████████   Terminal: konsole 
                               Terminal Font: Noto Mono 10 
                               CPU: Intel i9-10885H (16) @ 5.300GHz 
                               GPU: Intel CometLake-H GT2 [UHD Graphics] 
                               GPU: NVIDIA GeForce RTX 2060 Max-Q 
                               Memory: 11325MiB / 64060MiB

Inxi output:

System:
  Kernel: 5.13.4-1-MANJARO x86_64 bits: 64 compiler: gcc v: 11.1.0 
  parameters: BOOT_IMAGE=/boot/vmlinuz-5.13-x86_64 
  root=UUID=61563fba-524b-4fad-8d9e-d8b4c4bfde08 ro quiet apparmor=1 
  security=apparmor udev.log_priority=3 
  Desktop: KDE Plasma 5.22.3 tk: Qt 5.15.2 wm: kwin_x11 vt: 1 dm: SDDM 
  Distro: Manjaro Linux base: Arch Linux 
Machine:
  Type: Laptop System: Dell product: XPS 17 9700 v: N/A 
  serial: <superuser required> Chassis: type: 10 serial: <superuser required> 
  Mobo: Dell model: 03CPGC v: A00 serial: <superuser required> UEFI: Dell 
  v: 1.8.2 date: 05/21/2021 
Battery:
  ID-1: BAT0 charge: 34.5 Wh (43.8%) condition: 78.7/95.1 Wh (82.8%) 
  volts: 11.1 min: 11.4 model: BYD DELL F8CPG0B type: Li-poly serial: 221 
  status: Discharging 
Memory:
  RAM: total: 62.56 GiB used: 11.82 GiB (18.9%) 
  RAM Report: permissions: Unable to run dmidecode. Root privileges required. 
CPU:
  Info: 8-Core model: Intel Core i9-10885H bits: 64 type: MT MCP 
  arch: Comet Lake family: 6 model-id: A5 (165) stepping: 2 microcode: EA 
  cache: L2: 16 MiB bogomips: 76816 
  Speed: 2505 MHz min/max: 800/5300 MHz Core speeds (MHz): 1: 2505 2: 2588 
  3: 2579 4: 2527 5: 2502 6: 2557 7: 2489 8: 2493 9: 2532 10: 2511 11: 2523 
  12: 2538 13: 2521 14: 2379 15: 2521 16: 2533 
  Flags: 3dnowprefetch abm acpi adx aes aperfmperf apic arat arch_capabilities 
  arch_perfmon art avx avx2 bmi1 bmi2 bts clflush clflushopt cmov constant_tsc 
  cpuid cpuid_fault cx16 cx8 de ds_cpl dtes64 dtherm dts epb ept ept_ad erms 
  est f16c flexpriority flush_l1d fma fpu fsgsbase fxsr ht hwp hwp_act_window 
  hwp_epp hwp_notify ibpb ibrs ibrs_enhanced ida intel_pt invpcid 
  invpcid_single lahf_lm lm mca mce md_clear mmx monitor movbe mpx msr mtrr 
  nonstop_tsc nopl nx ospke pae pat pbe pcid pclmulqdq pdcm pdpe1gb pebs pge 
  pku pln pni popcnt pse pse36 pts rdrand rdseed rdtscp rep_good sdbg sep smap 
  smep smx ss ssbd sse sse2 sse4_1 sse4_2 ssse3 stibp syscall tm tm2 
  tpr_shadow tsc tsc_adjust tsc_deadline_timer vme vmx vnmi vpid x2apic 
  xgetbv1 xsave xsavec xsaveopt xsaves xtopology xtpr 
  Vulnerabilities: Type: itlb_multihit status: KVM: VMX disabled 
  Type: l1tf status: Not affected 
  Type: mds status: Not affected 
  Type: meltdown status: Not affected 
  Type: spec_store_bypass 
  mitigation: Speculative Store Bypass disabled via prctl and seccomp 
  Type: spectre_v1 
  mitigation: usercopy/swapgs barriers and __user pointer sanitization 
  Type: spectre_v2 mitigation: Enhanced IBRS, IBPB: conditional, RSB filling 
  Type: srbds status: Not affected 
  Type: tsx_async_abort status: Not affected 
Graphics:
  Device-1: Intel CometLake-H GT2 [UHD Graphics] vendor: Dell driver: i915 
  v: kernel bus-ID: 00:02.0 chip-ID: 8086:9bc4 class-ID: 0300 
  Device-2: NVIDIA TU106M [GeForce RTX 2060 Max-Q] vendor: Dell driver: nvidia 
  v: 470.57.02 alternate: nouveau,nvidia_drm bus-ID: 01:00.0 
  chip-ID: 10de:1f12 class-ID: 0300 
  Device-3: Realtek Integrated_Webcam_HD type: USB driver: uvcvideo 
  bus-ID: 1-5:3 chip-ID: 0bda:5510 class-ID: fe01 serial: 200901010001 
  Display: x11 server: X.Org 1.20.11 compositor: kwin_x11 driver: 
  loaded: modesetting,nvidia alternate: fbdev,intel,nouveau,nv,vesa 
  display-ID: :0 screens: 1 
  Screen-1: 0 s-res: 2560x1440 s-dpi: 96 s-size: 677x381mm (26.7x15.0") 
  s-diag: 777mm (30.6") 
  Monitor-1: eDP-1 res: 2560x1440 hz: 60 dpi: 178 size: 366x229mm (14.4x9.0") 
  diag: 432mm (17") 
  OpenGL: renderer: Mesa Intel UHD Graphics (CML GT2) v: 4.6 Mesa 21.1.5 
  direct render: Yes 
Audio:
  Device-1: Intel Comet Lake PCH cAVS vendor: Dell 
  driver: sof-audio-pci-intel-cnl 
  alternate: snd_hda_intel,snd_soc_skl,snd_sof_pci_intel_cnl bus-ID: 00:1f.3 
  chip-ID: 8086:06c8 class-ID: 0401 
  Sound Server-1: ALSA v: k5.13.4-1-MANJARO running: yes 
  Sound Server-2: JACK v: 0.125.0 running: no 
  Sound Server-3: PulseAudio v: 14.2 running: yes 
  Sound Server-4: PipeWire v: 0.3.32 running: yes 
Network:
  Device-1: Qualcomm QCA6390 Wireless Network Adapter [AX500-DBS ] 
  vendor: Rivet Networks driver: ath11k_pci v: kernel port: 3000 
  bus-ID: 04:00.0 chip-ID: 17cb:1101 class-ID: 0280 
  IF: wlp4s0 state: up mac: 9c:b6:d0:3e:f1:67 
  IP v4: 192.168.0.218/24 type: dynamic noprefixroute scope: global 
  broadcast: 192.168.0.255 
  IP v6: fd00:f0f2:49db:1a72:538d:8f40:c34b:108f/64 
  type: dynamic noprefixroute scope: global 
  IP v6: fe80::a013:d910:223d:468a/64 type: noprefixroute scope: link 
  IF-ID-1: br-18e56cd16eda state: down mac: 02:42:10:e8:43:90 
  IP v4: 172.23.0.1/16 scope: global broadcast: 172.23.255.255 
  IF-ID-2: br-5faaf5c30322 state: up speed: 10000 Mbps duplex: unknown 
  mac: 02:42:ef:53:f6:1c 
  IP v4: 172.21.0.1/16 scope: global broadcast: 172.21.255.255 
  IP v6: fe80::42:efff:fe53:f61c/64 scope: link 
  IF-ID-3: docker0 state: down mac: 02:42:0b:aa:c4:25 
  IP v4: 172.17.0.1/16 scope: global broadcast: 172.17.255.255 
  IP v6: fe80::42:bff:feaa:c425/64 scope: link 
  IF-ID-4: veth52aa549 state: up speed: 10000 Mbps duplex: full 
  mac: 52:38:78:e0:d5:c9 
  IF-ID-5: veth961b03c state: up speed: 10000 Mbps duplex: full 
  mac: 7a:a6:19:4d:45:1b 
  IF-ID-6: vethb1c3015 state: up speed: 10000 Mbps duplex: full 
  mac: 3a:31:32:6e:78:68 
  WAN IP: 89.99.94.174 
Bluetooth:
  Message: No bluetooth data found. 
Logical:
  Message: No logical block device data found. 
RAID:
  Message: No RAID data found. 
Drives:
  Local Storage: total: 1.86 TiB used: 289.74 GiB (15.2%) 
  SMART Message: Unable to run smartctl. Root privileges required. 
  ID-1: /dev/nvme0n1 maj-min: 259:0 vendor: KIOXIA 
  model: KXG60PNV2T04 NVMe 2048GB size: 1.86 TiB block-size: physical: 512 B 
  logical: 512 B speed: 31.6 Gb/s lanes: 4 type: SSD serial: 11JF30WWF9E3 
  rev: 10704104 temp: 37.9 C scheme: GPT 
  Message: No optical or floppy data found. 
Partition:
  ID-1: / raw-size: 1.43 TiB size: 1.41 TiB (98.36%) used: 289.65 GiB (20.1%) 
  fs: ext4 dev: /dev/nvme0n1p7 maj-min: 259:7 label: N/A 
  uuid: 61563fba-524b-4fad-8d9e-d8b4c4bfde08 
  ID-2: /boot/efi raw-size: 150 MiB size: 146 MiB (97.33%) 
  used: 92.9 MiB (63.6%) fs: vfat dev: /dev/nvme0n1p1 maj-min: 259:1 
  label: ESP uuid: 8EC7-22E3 
Swap:
  Alert: No swap data was found. 
Unmounted:
  ID-1: /dev/nvme0n1p2 maj-min: 259:2 size: 128 MiB fs: <superuser required> 
  label: N/A uuid: N/A 
  ID-2: /dev/nvme0n1p3 maj-min: 259:3 size: 423.79 GiB fs: bitlocker 
  label: N/A uuid: N/A 
  ID-3: /dev/nvme0n1p4 maj-min: 259:4 size: 990 MiB fs: ntfs label: WINRETOOLS 
  uuid: 5AE0CE44E0CE265B 
  ID-4: /dev/nvme0n1p5 maj-min: 259:5 size: 16.45 GiB fs: ntfs label: Image 
  uuid: 36FACEA4FACE5FAB 
  ID-5: /dev/nvme0n1p6 maj-min: 259:6 size: 1.4 GiB fs: ntfs 
  label: DELLSUPPORT uuid: 7CC47B47C47B02A6 
USB:
  Hub-1: 1-0:1 info: Full speed (or root) Hub ports: 16 rev: 2.0 
  speed: 480 Mb/s chip-ID: 1d6b:0002 class-ID: 0900 
  Device-1: 1-1:2 info: Shenzhen Goodix FingerPrint type: <vendor specific> 
  driver: N/A interfaces: 1 rev: 2.0 speed: 12 Mb/s power: 100mA 
  chip-ID: 27c6:533c class-ID: ff00 
  Device-2: 1-5:3 info: Realtek Integrated_Webcam_HD type: Video 
  driver: uvcvideo interfaces: 5 rev: 2.0 speed: 480 Mb/s power: 500mA 
  chip-ID: 0bda:5510 class-ID: fe01 serial: 200901010001 
  Hub-2: 2-0:1 info: Full speed (or root) Hub ports: 10 rev: 3.1 
  speed: 10 Gb/s chip-ID: 1d6b:0003 class-ID: 0900 
  Hub-3: 5-0:1 info: Full speed (or root) Hub ports: 2 rev: 2.0 
  speed: 480 Mb/s chip-ID: 1d6b:0002 class-ID: 0900 
  Hub-4: 6-0:1 info: Full speed (or root) Hub ports: 2 rev: 3.1 speed: 10 Gb/s 
  chip-ID: 1d6b:0003 class-ID: 0900 
  Hub-5: 7-0:1 info: Full speed (or root) Hub ports: 2 rev: 2.0 
  speed: 480 Mb/s chip-ID: 1d6b:0002 class-ID: 0900 
  Hub-6: 8-0:1 info: Full speed (or root) Hub ports: 2 rev: 3.1 speed: 10 Gb/s 
  chip-ID: 1d6b:0003 class-ID: 0900 
Sensors:
  System Temperatures: cpu: 50.0 C mobo: N/A 
  Fan Speeds (RPM): cpu: 1707 fan-2: 1702 
Info:
  Processes: 429 Uptime: 1h 9m wakeups: 7238 Init: systemd v: 248 
  tool: systemctl Compilers: gcc: 11.1.0 Packages: pacman: 1488 lib: 435 
  flatpak: 0 Shell: Zsh v: 5.8 running-in: konsole inxi: 3.3.06

When I reboot all is fine again until system.d freezes again. Never had this issue in the last 4 months running Manjaro.

Thank you for any insights.

2

Do you see the same problem when using another kernel (latest LTS 5.10 for example)?

3

The freeze just happened again after working without problems for the past 4 hours. I did not have the 5.10 kernel installed but did install it prior to the freeze. So I have now rebooted and using the 5.10 kernel and will continue working. Now just have to wait and see if the freeze happens. There is no way that I have found to reproduce the freeze at will.

Will keep you updated.

4

I’m having this exact same problem. I just upgraded earlier today.
Now whenever I use systemctl, systemd crashes.
The issue started happening after the update.

Broadcast message from systemd-journald@b0x (Mon 2021-07-26 20:15:59 +08):                               
                                                                                                         
systemd[1]: Caught <ABRT>, dumped core as pid 13243.                                                     
                                                                                                         
                                                                                                         
Broadcast message from systemd-journald@b0x (Mon 2021-07-26 20:15:59 +08):                               
                                                                                                         
systemd[1]: Freezing execution.

My Linux kernel is 5.4.134-1-MANJARO.

5

Not sure if this is related, but I’m also seeing systemd PID 1 dying in the last few days.

Looks like the same issue as reported and fixed here in Fedora: 1984651 – systemd[1]: Assertion 'a <= b' failed at src/libsystemd/sd-event/sd-event.c:2903, function sleep_between(). Aborting.

What I see in terms of logs:

Jul 26 22:31:19 x systemd[1]: Assertion 'a <= b' failed at src/libsystemd/sd-event/sd-event.c:2903, function sleep_between(). Aborting.
Jul 26 22:31:19 x audit[869041]: ANOM_ABEND auid=4294967295 uid=0 gid=0 ses=4294967295 pid=869041 comm="systemd" exe="/usr/lib/systemd/systemd" sig=6 res=1
Jul 26 22:31:19 x kernel: kauditd_printk_skb: 73 callbacks suppressed
Jul 26 22:31:19 x kernel: audit: type=1701 audit(1627335079.741:995): auid=4294967295 uid=0 gid=0 ses=4294967295 pid=869041 comm="systemd" exe="/usr/lib/systemd/systemd" sig=6 res=1
Jul 26 22:31:19 x systemd-coredump[869043]: Due to PID 1 having crashed coredump collection will now be turned off.
Jul 26 22:31:19 x systemd-coredump[869043]: [🡕] Process 869041 (systemd) of user 0 dumped core.
                                              
                                              Stack trace of thread 869041:
                                              #0  0x00007fc8fa55b01b kill (libc.so.6 + 0x3d01b)
                                              #1  0x000055fb302d4e97 n/a (systemd + 0x3ce97)
                                              #2  0x00007fc8fa6fd870 __restore_rt (libpthread.so.0 + 0x13870)
                                              #3  0x00007fc8fa55ad22 raise (libc.so.6 + 0x3cd22)
                                              #4  0x00007fc8fa544862 abort (libc.so.6 + 0x26862)
                                              #5  0x00007fc8fa8a2972 n/a (libsystemd-shared-248.so + 0x62972)
                                              #6  0x00007fc8faa41980 n/a (libsystemd-shared-248.so + 0x201980)
                                              #7  0x00007fc8faa4a03e n/a (libsystemd-shared-248.so + 0x20a03e)
                                              #8  0x00007fc8faa4aa8e sd_event_prepare (libsystemd-shared-248.so + 0x20aa8e)
                                              #9  0x00007fc8faa4dbb3 sd_event_run (libsystemd-shared-248.so + 0x20dbb3)
                                              #10 0x000055fb303195e8 n/a (systemd + 0x815e8)
                                              #11 0x000055fb302d1e2d n/a (systemd + 0x39e2d)
                                              #12 0x00007fc8fa545b25 __libc_start_main (libc.so.6 + 0x27b25)
                                              #13 0x000055fb302d4a0e n/a (systemd + 0x3ca0e)
Jul 26 22:31:19 x systemd[1]: Caught <ABRT>, dumped core as pid 869041.
Jul 26 22:31:19 x audit: BPF prog-id=26 op=UNLOAD
Jul 26 22:31:19 x systemd[1]: Freezing execution.
Jul 26 22:31:19 x kernel: audit: type=1334audit(1627335079.791:996): prog-id=26 op=UNLOAD
6

Related issue: Assertion 'a <= b' failed at src/libsystemd/sd-event/sd-event.c:2903, function sleep_between() · Issue #20285 · systemd/systemd · GitHub
Should have been fixed in upstream version 248.6.

7

yep, that’s the one. I’ll probably move over to the testing branch until this gets resolved in stable as my system is unusable for more than a few hours at a time.