Soft lock freeze breeze

[eviupol]

Hi,
for months now, every 2-5 days of uptime or so I get a nasty freeze.
I’ve enabled REISUB for such cases and it has allowed me to shut down the system in a “safer” way in many cases already but it doesn’t in this one.

sudo journalctl  -b -1 -e

Mar 07 23:19:30 eviu-PC kernel: watchdog: BUG: soft lockup - CPU#10 stuck for 23s! [khugepaged:122]
Mar 07 23:19:30 eviu-PC kernel: Modules linked in: vhost_net vhost vhost_iotlb tap tun ntfs3 snd_seq_dummy snd_seq_midi snd_hrtimer snd_seq_midi_event snd_seq xt_CH>
Mar 07 23:19:30 eviu-PC kernel:  acpi_pad acpi_tad mac_hid nct6687(OE) dm_multipath uinput crypto_user fuse bpf_preload ip_tables x_tables ext4 crc32c_generic crc16>
Mar 07 23:19:30 eviu-PC kernel: CPU: 10 PID: 122 Comm: khugepaged Tainted: G           OE      6.1.12-1-MANJARO #1 d419fb51ba9431ae2a4575820ea6b5b95f50a34f
Mar 07 23:19:30 eviu-PC kernel: Hardware name: Micro-Star International Co., Ltd. MS-7D25/PRO Z690-A DDR4(MS-7D25), BIOS 1.80 08/31/2022
Mar 07 23:19:30 eviu-PC kernel: RIP: 0010:smp_call_function_many_cond+0xee/0x310
Mar 07 23:19:30 eviu-PC kernel: Code: d0 48 89 df e8 93 47 40 00 3b 05 ed 94 ea 01 73 26 48 63 d0 49 8b 34 24 48 03 34 d5 a0 2a d6 ad 8b 56 08 83 e2 01 74 0a f3 90 >
Mar 07 23:19:30 eviu-PC kernel: RSP: 0018:ffffbf8440567c08 EFLAGS: 00000202
Mar 07 23:19:30 eviu-PC kernel: RAX: 0000000000000000 RBX: ffff9e7ba04b4108 RCX: 0000000000000001
Mar 07 23:19:30 eviu-PC kernel: RDX: 0000000000000001 RSI: ffffdf843f616a40 RDI: ffff9e7ba04b4108
Mar 07 23:19:30 eviu-PC kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: ffff9e7ba04a0ee0
Mar 07 23:19:30 eviu-PC kernel: R10: ffff9e6d561b5000 R11: 0000000000000000 R12: ffff9e7ba04b4100
Mar 07 23:19:30 eviu-PC kernel: R13: 0000000000000001 R14: 0000000000000010 R15: 000000000000000a
Mar 07 23:19:30 eviu-PC kernel: FS:  0000000000000000(0000) GS:ffff9e7ba0480000(0000) knlGS:0000000000000000
Mar 07 23:19:30 eviu-PC kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar 07 23:19:30 eviu-PC kernel: CR2: 00007f303deec000 CR3: 00000006299ae002 CR4: 0000000000f72ee0
Mar 07 23:19:30 eviu-PC kernel: PKRU: 55555554
Mar 07 23:19:30 eviu-PC kernel: Call Trace:
Mar 07 23:19:30 eviu-PC kernel:  <TASK>
Mar 07 23:19:30 eviu-PC kernel:  ? mm_take_all_locks+0x210/0x210
Mar 07 23:19:30 eviu-PC kernel:  smp_call_function+0x2c/0x50
Mar 07 23:19:30 eviu-PC kernel:  collapse_huge_page+0x5ba/0x1430
Mar 07 23:19:30 eviu-PC kernel:  ? schedule+0x5e/0xd0
Mar 07 23:19:30 eviu-PC kernel:  hpage_collapse_scan_pmd+0x5b2/0x820
Mar 07 23:19:30 eviu-PC kernel:  khugepaged+0x500/0x970
Mar 07 23:19:30 eviu-PC kernel:  ? collapse_pte_mapped_thp+0x5d0/0x5d0
Mar 07 23:19:30 eviu-PC kernel:  kthread+0xdb/0x110
Mar 07 23:19:30 eviu-PC kernel:  ? kthread_complete_and_exit+0x20/0x20
Mar 07 23:19:30 eviu-PC kernel:  ret_from_fork+0x1f/0x30
Mar 07 23:19:30 eviu-PC kernel:  </TASK>
Mar 07 23:19:58 eviu-PC kernel: watchdog: BUG: soft lockup - CPU#10 stuck for 49s! [khugepaged:122]

Non-standard stuff about my system:
pipewire
tkg-wine with fsync and yabridge for running audio plugins
RME FireFace 800 audio interface via LSI FW643 PCIe Firewire controller, the LSI card only shows up and works from a cold boot (not when I restart)
It also happened back on 5.19 kernels, now I’m on 6.1.12-1.

sysctl kernel.softlockup_panic kernel.hardlockup_panic

kernel.softlockup_panic = 0
kernel.hardlockup_panic = 0

The strange thing is, if I have htop open while this happens, it continues to update but mouse and keyboard are locked-up. Any video will lock up too.
If there was any audio playing, the fireface will repeat its last buffer continuously.
Please tell me if you need more information
If we can’t make the issue go away, the second-best would be to shut down the system safely to avoid file system corruption (also have some NTFS and LUKS+ext4 HDDs mounted via fstab).

[stephane]

can you report

inxi -Fza
sudo mhwd-kernel -li

[eviupol]

inxi

System:
  Kernel: 6.1.12-1-MANJARO arch: x86_64 bits: 64 compiler: gcc v: 12.2.1
    parameters: BOOT_IMAGE=/boot/vmlinuz-6.1-x86_64
    root=UUID=d1094b1b-69e9-4616-a17a-931c1219121b rw
    cryptdevice=UUID=917191e1-9111-418f-a191-1111112adaf0:luks-91719812-9111-4181-a191-51613c1a1a10
    root=/dev/mapper/luks-917191e1-9111-111f-a191-51111c1ad110 apparmor=1
    security=apparmor udev.log_priority=3
  Desktop: Xfce v: 4.18.1 tk: Gtk v: 3.24.36 info: xfce4-panel wm: xfwm
    v: 4.18.0 vt: 7 dm: LightDM v: 1.32.0 Distro: Manjaro Linux base: Arch Linux
Machine:
  Type: Desktop Mobo: Micro-Star model: PRO Z690-A DDR4(MS-7D25) v: 1.0
    serial: <superuser required> UEFI: American Megatrends LLC. v: 1.80
    date: 08/31/2022
CPU:
  Info: model: 12th Gen Intel Core i5-12600K bits: 64 type: MST AMCP
    arch: Alder Lake gen: core 12 level: v3 note: check built: 2021+
    process: Intel 7 (10nm ESF) family: 6 model-id: 0x97 (151) stepping: 2
    microcode: 0x26
  Topology: cpus: 1x cores: 10 mt: 6 tpc: 2 st: 4 threads: 16 smt: enabled
    cache: L1: 864 KiB desc: d-4x32 KiB, 6x48 KiB; i-6x32 KiB, 4x64 KiB
    L2: 9.5 MiB desc: 6x1.2 MiB, 1x2 MiB L3: 20 MiB desc: 1x20 MiB
  Speed (MHz): avg: 1547 high: 3700 min/max: 800/4900:3600 scaling:
    driver: intel_pstate governor: powersave cores: 1: 800 2: 3700 3: 800
    4: 3700 5: 800 6: 1269 7: 776 8: 3700 9: 766 10: 739 11: 3700 12: 815
    13: 800 14: 800 15: 800 16: 800 bogomips: 118000
  Flags: avx avx2 ht lm nx pae sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx
  Vulnerabilities:
  Type: itlb_multihit status: Not affected
  Type: l1tf status: Not affected
  Type: mds status: Not affected
  Type: meltdown status: Not affected
  Type: mmio_stale_data status: Not affected
  Type: retbleed status: Not affected
  Type: spec_store_bypass mitigation: Speculative Store Bypass disabled via
    prctl
  Type: spectre_v1 mitigation: usercopy/swapgs barriers and __user pointer
    sanitization
  Type: spectre_v2 mitigation: Enhanced IBRS, IBPB: conditional, RSB
    filling, PBRSB-eIBRS: SW sequence
  Type: srbds status: Not affected
  Type: tsx_async_abort status: Not affected
Graphics:
  Device-1: Intel AlderLake-S GT1 vendor: Micro-Star MSI driver: i915
    v: kernel ports: active: DP-1,HDMI-A-1 empty: DP-2, HDMI-A-2, HDMI-A-3,
    HDMI-A-4 bus-ID: 00:02.0 chip-ID: 8086:4680 class-ID: 0300
  Display: x11 server: X.Org v: 21.1.7 compositor: xfwm v: 4.18.0 driver: X:
    loaded: modesetting alternate: fbdev,vesa dri: iris gpu: i915
    display-ID: :0.0 screens: 1
  Screen-1: 0 s-res: 2400x1920 s-dpi: 96 s-size: 635x508mm (25.00x20.00")
    s-diag: 813mm (32.02")
  Monitor-1: DP-1 pos: right model: Dell U2412M serial: <filter> built: 2012
    res: 1200x1920 hz: 60 dpi: 94 gamma: 1.2 size: 324x518mm (12.76x20.39")
    diag: 611mm (24.1") ratio: 16:10 modes: max: 1920x1200 min: 720x400
  Monitor-2: HDMI-A-1 mapped: HDMI-1 pos: primary,left model: Dell U2412M
    serial: <filter> built: 2012 res: 1200x1920 hz: 60 dpi: 94 gamma: 1.2
    size: 324x518mm (12.76x20.39") diag: 611mm (24.1") ratio: 16:10 modes:
    max: 1920x1200 min: 720x400
  API: OpenGL v: 4.6 Mesa 22.3.5 renderer: Mesa Intel UHD Graphics 770
    (ADL-S GT1) direct-render: Yes
Audio:
  Device-1: HiBy FC3 type: USB driver: hid-generic,snd-usb-audio,usbhid
    bus-ID: 1-11.3:12 chip-ID: 32bb:0004 class-ID: 0300
  Sound API: ALSA v: k6.1.12-1-MANJARO running: yes
  Sound Server-1: PulseAudio v: 16.1 running: no
  Sound Server-2: PipeWire v: 0.3.65 running: yes
Network:
  Device-1: Intel Ethernet I225-V vendor: Micro-Star MSI driver: igc v: kernel
    pcie: gen: 2 speed: 5 GT/s lanes: 1 port: N/A bus-ID: 04:00.0
    chip-ID: 8086:15f3 class-ID: 0200
  IF: enp4s0 state: up speed: 1000 Mbps duplex: full mac: <filter>
  IF-ID-1: virbr0 state: up speed: 10 Mbps duplex: unknown mac: <filter>
  IF-ID-2: vnet0 state: unknown speed: 10 Mbps duplex: full mac: <filter>
Drives:
  Local Storage: total: 31.38 TiB used: 11.03 TiB (35.2%)
  SMART Message: Unable to run smartctl. Root privileges required.
  ID-1: /dev/nvme0n1 maj-min: 259:1 vendor: Samsung model: SSD 980 1TB
    size: 931.51 GiB block-size: physical: 512 B logical: 512 B speed: 31.6 Gb/s
    lanes: 4 type: SSD serial: <filter> rev: 2B4QFXO7 temp: 40.9 C scheme: GPT
  ID-2: /dev/nvme1n1 maj-min: 259:0 vendor: Samsung model: SSD 980 1TB
    size: 931.51 GiB block-size: physical: 512 B logical: 512 B speed: 31.6 Gb/s
    lanes: 4 type: SSD serial: <filter> rev: 2B4QFXO7 temp: 30.9 C scheme: GPT
  ID-3: /dev/sda maj-min: 8:0 vendor: Crucial model: CT500MX500SSD1
    size: 465.76 GiB block-size: physical: 4096 B logical: 512 B speed: 6.0 Gb/s
    type: SSD serial: <filter> rev: 033 scheme: GPT
  ID-4: /dev/sdb maj-min: 8:16 vendor: HGST (Hitachi) model: HMS5C4040BLE640
    size: 3.64 TiB block-size: physical: 4096 B logical: 512 B speed: 6.0 Gb/s
    type: HDD rpm: 5700 serial: <filter> rev: A5D0 scheme: GPT
  ID-5: /dev/sdc maj-min: 8:32 vendor: Western Digital
    model: WD40EZRZ-00GXCB0 size: 3.64 TiB block-size: physical: 4096 B
    logical: 512 B speed: 6.0 Gb/s type: HDD rpm: 5400 serial: <filter>
    rev: 0A80 scheme: GPT
  ID-6: /dev/sdd maj-min: 8:48 vendor: Western Digital
    model: WD121KRYZ-01W0RB0 size: 10.91 TiB block-size: physical: 4096 B
    logical: 512 B speed: 6.0 Gb/s type: HDD rpm: 7200 serial: <filter>
    rev: 1H01 scheme: GPT
  ID-7: /dev/sde maj-min: 8:64 vendor: Western Digital
    model: WD4002FYYZ-01B7CB0 size: 3.64 TiB block-size: physical: 512 B
    logical: 512 B speed: 6.0 Gb/s type: HDD rpm: 7200 serial: <filter>
    rev: 1M02 scheme: GPT
  ID-8: /dev/sdf maj-min: 8:80 vendor: Western Digital
    model: WD8004FRYZ-01VAEB0 size: 7.28 TiB block-size: physical: 4096 B
    logical: 512 B speed: 6.0 Gb/s type: HDD rpm: 7200 serial: <filter>
    rev: 1H01 scheme: GPT
Partition:
  ID-1: / raw-size: 931.21 GiB size: 915.52 GiB (98.32%)
    used: 225.47 GiB (24.6%) fs: ext4 dev: /dev/dm-0 maj-min: 254:0
    mapped: luks-11111112-9e11-1111-a111-511111111af0
  ID-2: /boot/efi raw-size: 300 MiB size: 299.4 MiB (99.80%)
    used: 484 KiB (0.2%) fs: vfat dev: /dev/nvme0n1p1 maj-min: 259:2
Swap:
  Alert: No swap data was found.
Sensors:
  System Temperatures: cpu: 35.5 C mobo: 41.0 C
  Fan Speeds (RPM): cpu: 332 mobo: 710
  Power: 12v: 12.10 5v: 4.97 3.3v: 3.32 vbat: N/A
Info:
  Processes: 444 Uptime: 12h 44m wakeups: 0 Memory: 78.37 GiB
  used: 24.67 GiB (31.5%) Init: systemd v: 252 default: graphical
  tool: systemctl Compilers: gcc: 12.2.1 clang: 15.0.7 Packages: pm: pacman
  pkgs: 1986 libs: 500 tools: pamac pm: flatpak pkgs: 0 Shell: Bash v: 5.1.16
  running-in: xfce4-terminal inxi: 3.3.25

mhwd

Currently running: 6.1.12-1-MANJARO (linux61)
The following kernels are installed in your system:
   * linux519
   * linux61

[varikonniemi]

just a random thought try linux62 to rule that out?

[stephane]

can you add on boot
“ibt=off” and remove linux519 , add linux62

see this

Changing kernel version causes boot hang during "audit" - Intel 12 & 13 gen

can you please add on boot “ibt=off” you can add on grub wit (e)dit command ( keyboard is in Qwerty ) or add a this line GRUB_CMDLINE_LINUX_DEFAULT= sudo nano /etc/default/grub ( ctrl + X for confirm save Y/N ) sudo update-grub

sudo mhwd-kernel -r linux519 
sudo mhwd-kernel -i linux62