Total system hangs and fs/iomap kernel panic on 7.0.10 (Stable on 6.18 LTS)

I am experiencing severe system instability and total hard-locks after upgrading to the 7.0.10-1-MANJARO kernel. The system becomes sluggish, eventually freezing completely. During these hangs, I cannot even drop to a TTY, forcing a hard reset. Also, hibernation was also not working.

Rolling back to the 6.18 LTS kernel completely resolves all issues, and the system is rock-solid.

journalctl -b -1 -p 3 -x shows:

Jun 12 01:42:39 shaurya-manjaro kernel: amdgpu 0000:11:00.0: Unable to locate a BIOS ROM
Jun 12 01:42:39 shaurya-manjaro kernel: amdgpu 0000:11:00.0: Fatal error during GPU init
Jun 12 01:42:39 shaurya-manjaro kernel: amdgpu 0000:11:00.0: GPU mode2 reset failed
Jun 12 01:42:39 shaurya-manjaro kernel: amdgpu 0000:11:00.0: asic reset on amdgpu_device_ip_fini_early failed
Jun 12 01:42:39 shaurya-manjaro kernel: amdgpu 0000:11:00.0: probe with driver amdgpu failed with error -22

Further down the line there is also:

Jun 12 01:56:38 shaurya-manjaro kernel: kernel BUG at fs/iomap/buffered-io.c:1061!

On a bit of searching, the amdgpu bug seems to resolve by adding amdgpu.sg_display=0 to GRUB configuration, as I have not seen the same error in journalctl since.

However, for the kernel bug, I found similar complaints in arch forum and kernel bugtracker. Reading the bug report in kernel bug tracker, it seems nothing is being done for it. I also found a commit for the same in cachy os repo for the ntfs3 driver. So is there plan to patch the manjaro kernel too?

Forgot to add inxi output:

System:
  Kernel: 7.0.10-1-MANJARO arch: x86_64 bits: 64 compiler: gcc v: 16.1.1
    clocksource: tsc avail: hpet,acpi_pm
    parameters: BOOT_IMAGE=/@/boot/vmlinuz-7.0-x86_64
    root=UUID=cdb11ae4-4e46-4fc9-93ef-252d44f2ef33 rw rootflags=subvol=@ quiet
    apparmor=1 security=apparmor
    resume=UUID=082584e8-0acf-4296-8a3b-323e927ab602 udev.log_priority=3
    transparent_hugepage=madvise
  Desktop: KDE Plasma v: 6.6.5 tk: Qt v: N/A info: frameworks v: 6.26.0
    wm: kwin_wayland vt: 2 dm: SDDM Distro: Manjaro base: Arch Linux
Machine:
  Type: Desktop Mobo: Micro-Star model: B650 GAMING PLUS WIFI (MS-7E26) v: 1.0
    serial: <superuser required> Firmware: UEFI-[Legacy] vendor: American
    Megatrends LLC. v: 1.G2 date: 08/29/2024
CPU:
  Info: model: AMD Ryzen 7 9700X bits: 64 type: MT MCP arch: Zen 5 gen: 5
    level: v4 note: check built: 2024+ process: TSMC n4 (4nm) family: 0x1A (26)
    model-id: 0x44 (68) stepping: 0 microcode: 0xB40401C
  Topology: cpus: 1x dies: 1 clusters: 1 cores: 8 threads: 16 tpc: 2
    smt: enabled cache: L1: 640 KiB desc: d-8x48 KiB; i-8x32 KiB L2: 8 MiB
    desc: 8x1024 KiB L3: 32 MiB desc: 1x32 MiB
  Speed (MHz): avg: 3784 min/max: 605/5582 boost: enabled scaling:
    driver: amd-pstate-epp governor: powersave cores: 1: 3784 2: 3784 3: 3784
    4: 3784 5: 3784 6: 3784 7: 3784 8: 3784 9: 3784 10: 3784 11: 3784 12: 3784
    13: 3784 14: 3784 15: 3784 16: 3784 bogomips: 121375
  Flags-basic: avx avx2 ht lm nx pae sse sse2 sse3 sse4_1 sse4_2 sse4a ssse3
    svm
  Vulnerabilities:
  Type: gather_data_sampling status: Not affected
  Type: ghostwrite status: Not affected
  Type: indirect_target_selection status: Not affected
  Type: itlb_multihit status: Not affected
  Type: l1tf status: Not affected
  Type: mds status: Not affected
  Type: meltdown status: Not affected
  Type: mmio_stale_data status: Not affected
  Type: old_microcode status: Not affected
  Type: reg_file_data_sampling status: Not affected
  Type: retbleed status: Not affected
  Type: spec_rstack_overflow mitigation: IBPB on VMEXIT only
  Type: spec_store_bypass mitigation: Speculative Store Bypass disabled via
    prctl
  Type: spectre_v1 mitigation: usercopy/swapgs barriers and __user pointer
    sanitization
  Type: spectre_v2 mitigation: Enhanced / Automatic IBRS; IBPB: conditional;
    STIBP: always-on; PBRSB-eIBRS: Not affected; BHI: Not affected
  Type: srbds status: Not affected
  Type: tsa status: Not affected
  Type: tsx_async_abort status: Not affected
  Type: vmscape mitigation: IBPB on VMEXIT
Graphics:
  Device-1: Advanced Micro Devices [AMD/ATI] Navi 32 [Radeon RX 7700 XT / 7800
    XT] driver: amdgpu v: kernel arch: RDNA-3 code: Navi-3x
    process: TSMC n5 (5nm) built: 2022+ pcie: gen: 4 speed: 16 GT/s lanes: 16
    ports: active: DP-2 empty: DP-1, DP-3, HDMI-A-1, Writeback-1
    bus-ID: 03:00.0 chip-ID: 1002:747e class-ID: 0300
  Device-2: Advanced Micro Devices [AMD/ATI] Granite Ridge [Radeon Graphics]
    vendor: Micro-Star MSI driver: N/A alternate: amdgpu arch: RDNA-2
    code: Navi-2x process: TSMC n7 (7nm) built: 2020-22 pcie: gen: 4
    speed: 16 GT/s lanes: 16 bus-ID: 11:00.0 chip-ID: 1002:13c0 class-ID: 0300
  Device-3: Lenovo FHD Webcam driver: snd-usb-audio,uvcvideo type: USB
    rev: 2.0 speed: 480 Mb/s lanes: 1 mode: 2.0 bus-ID: 1-5.2:4
    chip-ID: 17ef:4831 class-ID: 0102 serial: <filter>
  Device-4: MACROSILICON USB3.0 Capture
    driver: hid-generic,snd-usb-audio,usbhid,uvcvideo type: USB rev: 3.2
    speed: 5 Gb/s lanes: 1 mode: 3.2 gen-1x1 bus-ID: 2-2:2 chip-ID: 345f:2130
    class-ID: 0300 serial: <filter>
  Display: wayland server: X.org v: 1.21.1.22 with: Xwayland v: 24.1.11
    compositor: kwin_wayland driver: X: loaded: amdgpu
    unloaded: modesetting,radeon alternate: fbdev,vesa dri: radeonsi
    gpu: amdgpu display-ID: 0
  Monitor-1: DP-2 model: AOC 2260WG5 serial: <filter> built: 2015 res:
    mode: 1920x1080 hz: 75 scale: 100% (1) dpi: 102 gamma: 1.2
    size: 477x268mm (18.78x10.55") diag: 547mm (21.5") ratio: 16:9 modes:
    max: 1920x1080 min: 720x400
  API: EGL v: 1.5 hw: drv: amd radeonsi platforms: device: 0 drv: radeonsi
    device: 1 drv: swrast gbm: drv: radeonsi surfaceless: drv: radeonsi wayland:
    drv: radeonsi x11: drv: radeonsi
  API: OpenGL v: 4.6 vendor: amd mesa v: 26.1.1-arch1.2 glx-v: 1.4
    direct-render: yes renderer: AMD Radeon RX 7800 XT (radeonsi navi32 ACO DRM
    3.64 7.0.10-1-MANJARO) device-ID: 1002:747e memory: 15.62 GiB unified: no
    display-ID: :0.0
  API: Vulkan v: 1.4.350 layers: 10 device: 0 type: discrete-gpu name: AMD
    Radeon RX 7800 XT (RADV NAVI32) driver: mesa radv v: 26.1.1-arch1.2
    device-ID: 1002:747e surfaces: N/A device: 1 type: cpu name: llvmpipe
    (LLVM 22.1.5 256 bits) driver: mesa llvmpipe v: 26.1.1-arch1.2 (LLVM
    22.1.5) device-ID: 10005:0000 surfaces: N/A
  Info: Tools: api: clinfo, eglinfo, glxinfo, vulkaninfo
    de: kscreen-console,kscreen-doctor gpu: amdgpu_top, corectrl, lact, radeontop
    wl: wayland-info x11: xdpyinfo, xprop, xrandr
Audio:
  Device-1: Advanced Micro Devices [AMD/ATI] Navi 31 HDMI/DP Audio
    driver: snd_hda_intel v: kernel pcie: gen: 4 speed: 16 GT/s lanes: 16
    bus-ID: 03:00.1 chip-ID: 1002:ab30 class-ID: 0403
  Device-2: Advanced Micro Devices [AMD/ATI] Radeon High Definition Audio
    vendor: Micro-Star MSI driver: snd_hda_intel v: kernel pcie: gen: 4
    speed: 16 GT/s lanes: 16 bus-ID: 11:00.1 chip-ID: 1002:1640 class-ID: 0403
  Device-3: Advanced Micro Devices [AMD] Ryzen HD Audio
    vendor: Micro-Star MSI driver: snd_hda_intel v: kernel pcie: gen: 4
    speed: 16 GT/s lanes: 16 bus-ID: 11:00.6 chip-ID: 1022:15e3 class-ID: 0403
  Device-4: Lenovo FHD Webcam driver: snd-usb-audio,uvcvideo type: USB
    rev: 2.0 speed: 480 Mb/s lanes: 1 mode: 2.0 bus-ID: 1-5.2:4
    chip-ID: 17ef:4831 class-ID: 0102 serial: <filter>
  Device-5: Logitech G435 Wireless Gaming Headset
    driver: hid-generic,snd-usb-audio,usbhid type: USB rev: 2.0 speed: 12 Mb/s
    lanes: 1 mode: 1.1 bus-ID: 1-9:8 chip-ID: 046d:0adf class-ID: 0300
    serial: <filter>
  Device-6: MACROSILICON USB3.0 Capture
    driver: hid-generic,snd-usb-audio,usbhid,uvcvideo type: USB rev: 3.2
    speed: 5 Gb/s lanes: 1 mode: 3.2 gen-1x1 bus-ID: 2-2:2 chip-ID: 345f:2130
    class-ID: 0300 serial: <filter>
  API: ALSA v: k7.0.10-1-MANJARO status: kernel-api with: aoss
    type: oss-emulator tools: alsactl,alsamixer,amixer
  Server-1: sndiod v: N/A status: off tools: aucat,midicat,sndioctl
  Server-2: PipeWire v: 1.6.5 status: active with: 1: pipewire-pulse
    status: active 2: wireplumber status: active 3: pipewire-alsa type: plugin
    4: pw-jack type: plugin tools: pactl,pw-cat,pw-cli,wpctl
Network:
  Device-1: Realtek RTL8125 2.5GbE vendor: Micro-Star MSI driver: r8169
    v: kernel pcie: gen: 2 speed: 5 GT/s lanes: 1 port: e000 bus-ID: 0d:00.0
    chip-ID: 10ec:8125 class-ID: 0200
  IF: enp13s0 state: up speed: 1000 Mbps duplex: full mac: <filter>
  Device-2: MEDIATEK MT7922 802.11ax PCI Express Wireless Network Adapter
    driver: mt7921e v: kernel pcie: gen: 2 speed: 5 GT/s lanes: 1 bus-ID: 0e:00.0
    chip-ID: 14c3:0616 class-ID: 0280
  IF: wlp14s0 state: up mac: <filter>
  Info: services: NetworkManager, systemd-timesyncd, wpa_supplicant
Bluetooth:
  Device-1: MediaTek Wireless_Device driver: btusb v: 0.8 type: USB rev: 2.1
    speed: 480 Mb/s lanes: 1 mode: 2.0 bus-ID: 1-7:5 chip-ID: 0e8d:0616
    class-ID: e001 serial: <filter>
  Report: rfkill ID: hci0 rfk-id: 0 state: up address: see --recommends
Drives:
  Local Storage: total: 912.89 GiB used: 200.59 GiB (22.0%)
  SMART Message: Unable to run smartctl. Root privileges required.
  ID-1: /dev/sda maj-min: 8:0 vendor: Crucial model: CT480BX500SSD1
    size: 447.13 GiB block-size: physical: 512 B logical: 512 B speed: 6.0 Gb/s
    tech: SSD serial: <filter> fw-rev: 041 scheme: GPT
  ID-2: /dev/sdb maj-min: 8:16 vendor: Western Digital
    model: WD5000AAKX-001CA0 size: 465.76 GiB block-size: physical: 512 B
    logical: 512 B speed: 6.0 Gb/s tech: N/A serial: <filter> fw-rev: 1H15
    scheme: MBR
Partition:
  ID-1: / raw-size: 240 GiB size: 240 GiB (100.00%) used: 200.59 GiB (83.6%)
    fs: btrfs dev: /dev/sda2 maj-min: 8:2
  ID-2: /home raw-size: 240 GiB size: 240 GiB (100.00%)
    used: 200.59 GiB (83.6%) fs: btrfs dev: /dev/sda2 maj-min: 8:2
  ID-3: /var/log raw-size: 240 GiB size: 240 GiB (100.00%)
    used: 200.59 GiB (83.6%) fs: btrfs dev: /dev/sda2 maj-min: 8:2
Swap:
  Kernel: swappiness: 60 (default) cache-pressure: 100 (default) zswap: yes
    compressor: zstd max-pool: 20%
  ID-1: swap-1 type: partition size: 9 GiB used: 0 KiB (0.0%) priority: -1
    dev: /dev/sda3 maj-min: 8:3
Sensors:
  System Temperatures: cpu: 46.0 C mobo: 45.0 C gpu: amdgpu temp: 61.0 C
    mem: 68.0 C
  Fan Speeds (rpm): cpu: 626 mobo: 575 gpu: amdgpu fan: 531
  Power: 12v: 12.00 5v: 4.97 3.3v: 3.34 vbat: N/A gpu: amdgpu watts: 46.00
Info:
  Memory: total: 32 GiB note: est. available: 30.45 GiB used: 4.89 GiB (16.1%)
  Processes: 377 Power: uptime: 49m states: freeze,mem,disk suspend: deep
    avail: s2idle wakeups: 0 hibernate: platform avail: shutdown, reboot,
    suspend, test_resume image: 12.15 GiB services: org_kde_powerdevil,upowerd
    Init: systemd v: 260 default: graphical tool: systemctl
  Packages: pm: dpkg pkgs: 0 pm: pacman pkgs: 2797 libs: 563 tools: pamac
    pm: flatpak pkgs: 0 Compilers: clang: 22.1.5 gcc: 16.1.1 Shell: Zsh v: 5.9
    running-in: yakuake inxi: 3.3.40

I would keep 6.18 as the working boot entry for now and treat 7.0.10 as unsafe on that machine until the ntfs3/iomap fix lands. If the NTFS volume is not essential at boot, one practical test is to leave it unmounted for a session on 7.0.10 and see if the panic disappears; if it does, mount it via ntfs-3g or stay on LTS for that workload.

Also keep a separate backup before testing hibernation again. A hard lock while ntfs3 is writing is exactly the kind of failure where I would rather be boring and use the stable kernel than chase a newer one.

1 Like

I am a community member just like yourself and no kernel maintainer.

kernel on manjaro

Manjaro kernels are build from upstream source https://kernel.org

Manjaro kernel maintainer maintains a set of patches targeting specific hardware.

You can find these at core packages repo see the specific kernel repositories e.g. linux618, linux70 and linux71 - please consult the PKGBUILD to see which patches are actually applied (unused patches are kept for reference).

If this question relates to fix ntfs, I think kernel maintainer will say no.

ntfs is a proprietary alien file system and the kernel driver is reverse engineered and it is far from perfect.

Any reliance on ntfs is - forgive me for being blunt - not a Manjaro issue.

if you must have ntfs

Blacklist the kernel driver to prioritise the user-space ntfs-3g driver.

If your issue is related to boot because you mount the device in your fstab I advise to change your approach to use a set of systemd units (mount/automount) to mount the device when path is accessed.

More information on the use of systemd mount/automount units can be found in Contributions > Tutorials

[root tip] [How To] systemd mount unit samples

[root tip] [How To] Use systemd to mount ANY device

2 Likes

You may need to remind yourself of the Cachy OS repository policy.

Repository Usage Policy | CachyOS

1 Like

@linux-aarhus
I am an individual and don’t distribute manjaro or any other distro in any official or non-official capacity, so i don’t see any issue. Their usage policy may apply to you though, if you included their compiled kernel in manjaro or some official distribution channel. It still does not bar the manjaro team from using their patches though

@giulia_privacy408
Accidentally deleted the previous comment while editing, so posting again.

Yeah, I am back to 6.18 kernel. Besides the ntfs issue, which i solved with first compiling my own kernel with the patch (and later just using the cachy-os kernel), and the amd bug mentioned above, there is another amdgpu bug where a null pointer deference freezes the system, and which I can not solve. Posted it in amdgpu gitlab , but i have seen little activity to show it will addressed in near future.

@linux-aarhus
Btw, manjaro team should really look into this ntfs bug as it is not hardware specific and will likely affect any user using the said driver in kernel 7.0.x. So either they should blacklist the newer ntfs driver and make ntfs-3g the default, or include the patch in the recent kernels. It can cause data corruption if triggered at wrong time.

Mod edit: Consecutive posts merged, @mentions added.

Nonetheless - the policy seems very clear on the subject - that is the reason I mentioned the policy

4. Supported Users

The CachyOS repository (and its mirrors) provides support exclusively for the following users:

  • CachyOS Users
  • ArchLinux Users

5. Unsupported Users

Users from other distributions are NOT SUPPORTED and HIGHLY DISCOURAGED to use the CachyOS repository (and its mirrors). This includes, but is not limited to:

  • Manjaro
  • EndeavourOS
  • ArcoLinux
  • Parabola
  • Users of any other Linux distribution not explicitly mentioned in the “Supported Users” section.

While you could read it as you do - but the words - appears to be valid for any individual that might get the idea of mixing in cachyos packages

Users from other distributions are NOT SUPPORTED and HIGHLY DISCOURAGED to use the CachyOS repository (and its mirrors).

But it is your system - you can do whatever you like.

As for ntfs if it really is as serious as you say - it is upstream at kernel.org that needs to fix it - not individual distributions.

2 Likes

I don’t need their support. I know the risk i am taking. It was just more convenient than compiling my own kernel

However, as much as CachyOS do have their policy, so does Manjaro – technically you would be in an unsupported state – no Support should be offered.

This was indeed a wise choice.

2 Likes

As noted ntfs kernel driver is a matter for upstream

Linux 7.1 has done exactly that.

https://forum.manjaro.org/t/linux-kernel-7-1-released-with-rewritten-ntfs-support/188331

The -rc7 is still the latest in unstable branch - I am sure it will become available very soon.