After monday update I can't boot with 6.x kernel

Hello. Don’t know what happened but I updated on monday and also yesterday. Since then I have to choose only manjaro with kernel 5.x (5.15.113-1). I always keep old if something like this happens. With kernel 6.x (even LTS - 6.1) I’m stuck on starting services for ~8 minutes and then they fail and nothing happens. See pictures. All is good on 5.x.
https://paste.pics/5d84518e9090a0134cadb9fcf6a21178
https://paste.pics/a09aad5579f910f3950987a487118fff
I have no idea what is wrong I don’t see anything bad in logs because they looks like wiped after I sucessfully boot 5.x kernel
Here is all info from manjaro-log
https://0x0.st/Hbbw.txt

Edit: Since monday I was running on 6.x LTS without problems

were the kernels working before the update?
post output from:
mhwd-kernel -li && mhwd -l -li

Yes the 6 kernel was working before. Had no problem until monday

Currently running: 5.15.113-1-MANJARO (linux515)
The following kernels are installed in your system:
   * linux515
   * linux61
> Installed PCI configs:
--------------------------------------------------------------------------------
                  NAME               VERSION          FREEDRIVER           TYPE
--------------------------------------------------------------------------------
           video-linux            2018.05.04                true            PCI
         network-r8168            2016.04.20                true            PCI


Warning: No installed USB configs!
> 0000:01:00.0 (0200:10ec:8168) Network controller Realtek Semiconductor Co., Ltd.:
--------------------------------------------------------------------------------
                  NAME               VERSION          FREEDRIVER           TYPE
--------------------------------------------------------------------------------
         network-r8168            2016.04.20                true            PCI


> 0000:00:02.0 (0300:8086:5917) Display controller Intel Corporation:
--------------------------------------------------------------------------------
                  NAME               VERSION          FREEDRIVER           TYPE
--------------------------------------------------------------------------------
           video-linux            2018.05.04                true            PCI
     video-modesetting            2020.01.13                true            PCI
            video-vesa            2017.03.12                true            PCI

Please check the `/var/log/pacman.log´ for updates you did on Monday. Older kernels you may find here:

This is what I found I was upgrading from 6.3.3-1 to 6.3.4-1 also 6.1.29-1 to 6.1.30-1
On thursday it was from 6.3.4-1 to 6.3.4-2 because I thought it has fix.
Both of them are currently not working. I removed 6.3.4-2 because I was trying to fix it but it didn’t helped
https://0x0.st/Hbc-.txt

You have three kernels installed. 5.15, 6.1 and 6.3. Which of those series were the last working for you? 6.2.4-2 only fixes an issue with XFS filesystem. You can try to install the kernels built yesterday and today with their extramodules.

If you know the last working kernel version we might be able to see what is missing on your end.

It seems, I run into the same issue on a laptop with optimus (Intel + Nvidia-GPU).
After a fresh install (kernel 6.1.29) everything was fine but after update to kernel version 6.1.30 I can’t login anymore. I did another fresh install and installed kernel 5.10 before I updated anything and so I can use this one.
This should also effect other arch based distros and kernel 6.x. Same thing happens with endeavouros kernel 6.1 and 6.3. With endeavouros I did not install a desktop environment. The systems hangs as soon as I tried to login to a console.

The corresponding log entries (journalctl) could be:

Mai 30 20:41:07 ConceptD7 sddm[641]: Greeter session started successfully
Mai 30 20:41:07 ConceptD7 sddm-greeter[690]: High-DPI autoscaling not Enabled
Mai 30 20:41:07 ConceptD7 sddm-greeter[690]: Reading from "/usr/share/wayland-sessions/plasmawayland.desktop"
Mai 30 20:41:07 ConceptD7 sddm-greeter[690]: Reading from "/usr/share/xsessions/plasma.desktop"
Mai 30 20:41:07 ConceptD7 sddm-greeter[690]: Loading theme configuration from "/usr/share/sddm/themes/breath/theme.conf"
Mai 30 20:41:07 ConceptD7 systemd[682]: Created slice User Core Session Slice.
Mai 30 20:41:07 ConceptD7 systemd[682]: Starting D-Bus User Message Bus...
Mai 30 20:41:07 ConceptD7 systemd[682]: Started D-Bus User Message Bus.
Mai 30 20:41:07 ConceptD7 sddm-greeter[690]: Connected to the daemon.
Mai 30 20:41:07 ConceptD7 sddm-greeter[690]: QFont::fromString: Invalid description '(empty)'
Mai 30 20:41:07 ConceptD7 sddm[641]: Message received from greeter: Connect
Mai 30 20:41:07 ConceptD7 sddm-greeter[690]: Loading file:///usr/share/sddm/themes/breath/Main.qml...
Mai 30 20:41:07 ConceptD7 kernel: BUG: kernel NULL pointer dereference, address: 0000000000000000
Mai 30 20:41:07 ConceptD7 kernel: #PF: supervisor read access in kernel mode
Mai 30 20:41:07 ConceptD7 kernel: #PF: error_code(0x0000) - not-present page
Mai 30 20:41:07 ConceptD7 kernel: PGD 0 P4D 0 
Mai 30 20:41:07 ConceptD7 kernel: Oops: 0000 [#1] PREEMPT SMP PTI
Mai 30 20:41:07 ConceptD7 kernel: CPU: 7 PID: 129 Comm: kworker/7:1 Tainted: P           OE      6.1.30-1-MANJARO #1 6d2f515ee684a660a7ad955d671d274ab26d4d3b
Mai 30 20:41:07 ConceptD7 kernel: Hardware name: Acer ConceptD CN715-71/GhibliC_CFS, BIOS V1.09 09/15/2020
Mai 30 20:41:07 ConceptD7 kernel: Workqueue: events_long ucsi_init_work [typec_ucsi]
Mai 30 20:41:07 ConceptD7 kernel: RIP: 0010:ucsi_acpi_async_write+0x30/0x50 [ucsi_acpi]
Mai 30 20:41:07 ConceptD7 kernel: Code: 44 00 00 41 55 49 89 cd 41 54 49 89 d4 55 53 89 f3 e8 24 fe fb ff 4c 89 e6 89 df 4c 89 ea 48 03 78 10 48 89 c5 e8 d0 ff 12 d4 <49> 8b 04 24 48 89 ef be 01 00 00 00 48 89 45 50 5b 5d 41 5c 41 5d
Mai 30 20:41:07 ConceptD7 kernel: RSP: 0018:ffffb2a4c05f7db0 EFLAGS: 00010282
Mai 30 20:41:07 ConceptD7 kernel: RAX: ffffb2a4c018d002 RBX: 0000000000000002 RCX: 0000000000000000
Mai 30 20:41:07 ConceptD7 kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffb2a4c018d002
Mai 30 20:41:07 ConceptD7 kernel: RBP: ffff89d08132a4a8 R08: 0000000000000000 R09: ffff89d09c944d60
Mai 30 20:41:07 ConceptD7 kernel: R10: ffffffff94f4bf80 R11: 0000000000000001 R12: 0000000000000000
Mai 30 20:41:07 ConceptD7 kernel: R13: 0000000000000000 R14: ffff89d08c1f7eb8 R15: 0000000000000000
Mai 30 20:41:07 ConceptD7 kernel: FS:  0000000000000000(0000) GS:ffff89d7f43c0000(0000) knlGS:0000000000000000
Mai 30 20:41:07 ConceptD7 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mai 30 20:41:07 ConceptD7 kernel: CR2: 0000000000000000 CR3: 00000004c2610001 CR4: 00000000003706e0
Mai 30 20:41:07 ConceptD7 kernel: Call Trace:
Mai 30 20:41:07 ConceptD7 kernel:  <TASK>
Mai 30 20:41:07 ConceptD7 kernel:  ucsi_exec_command+0x1e6/0x200 [typec_ucsi 5a73f76ef4f66e47c061ea35ea2f81bb41592690]
Mai 30 20:41:07 ConceptD7 kernel:  ucsi_send_command+0x4b/0xe0 [typec_ucsi 5a73f76ef4f66e47c061ea35ea2f81bb41592690]
Mai 30 20:41:07 ConceptD7 kernel:  ucsi_init_work+0x337/0x720 [typec_ucsi 5a73f76ef4f66e47c061ea35ea2f81bb41592690]
Mai 30 20:41:07 ConceptD7 kernel:  process_one_work+0x1c4/0x380
Mai 30 20:41:07 ConceptD7 kernel:  worker_thread+0x51/0x390
Mai 30 20:41:07 ConceptD7 kernel:  ? rescuer_thread+0x3b0/0x3b0
Mai 30 20:41:07 ConceptD7 kernel:  kthread+0xdb/0x110
Mai 30 20:41:07 ConceptD7 kernel:  ? kthread_complete_and_exit+0x20/0x20
Mai 30 20:41:07 ConceptD7 kernel:  ret_from_fork+0x1f/0x30
Mai 30 20:41:07 ConceptD7 kernel:  </TASK>
Mai 30 20:41:07 ConceptD7 kernel: Modules linked in: qrtr cmac algif_hash algif_skcipher af_alg bnep nvidia_uvm(POE) nvidia_drm(POE) snd_sof_pci_intel_cnl snd_sof_intel_hda_common soundwire_intel soundwire_generic_allocation soundwire_>
Mai 30 20:41:07 ConceptD7 kernel:  snd_intel_sdw_acpi crypto_simd btmtk cryptd libcrc32c fat snd_hda_codec videobuf2_vmalloc processor_thermal_rfim rapl videobuf2_memops bluetooth iTCO_wdt acer_wmi snd_hda_core drm_buddy processor_ther>
Mai 30 20:41:07 ConceptD7 kernel:  atkbd libps2 vivaldi_fmap nvme nvme_core crc32c_intel spi_intel_pci xhci_pci spi_intel nvme_common xhci_pci_renesas i8042 serio
Mai 30 20:41:07 ConceptD7 kernel: CR2: 0000000000000000
Mai 30 20:41:07 ConceptD7 kernel: ---[ end trace 0000000000000000 ]---
Mai 30 20:41:07 ConceptD7 kernel: RIP: 0010:ucsi_acpi_async_write+0x30/0x50 [ucsi_acpi]
Mai 30 20:41:07 ConceptD7 kernel: Code: 44 00 00 41 55 49 89 cd 41 54 49 89 d4 55 53 89 f3 e8 24 fe fb ff 4c 89 e6 89 df 4c 89 ea 48 03 78 10 48 89 c5 e8 d0 ff 12 d4 <49> 8b 04 24 48 89 ef be 01 00 00 00 48 89 45 50 5b 5d 41 5c 41 5d
Mai 30 20:41:07 ConceptD7 kernel: RSP: 0018:ffffb2a4c05f7db0 EFLAGS: 00010282
Mai 30 20:41:07 ConceptD7 kernel: RAX: ffffb2a4c018d002 RBX: 0000000000000002 RCX: 0000000000000000
Mai 30 20:41:07 ConceptD7 kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffb2a4c018d002
Mai 30 20:41:07 ConceptD7 kernel: RBP: ffff89d08132a4a8 R08: 0000000000000000 R09: ffff89d09c944d60
Mai 30 20:41:07 ConceptD7 kernel: R10: ffffffff94f4bf80 R11: 0000000000000001 R12: 0000000000000000
Mai 30 20:41:07 ConceptD7 kernel: R13: 0000000000000000 R14: ffff89d08c1f7eb8 R15: 0000000000000000
Mai 30 20:41:07 ConceptD7 kernel: FS:  0000000000000000(0000) GS:ffff89d7f43c0000(0000) knlGS:0000000000000000
Mai 30 20:41:07 ConceptD7 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mai 30 20:41:07 ConceptD7 kernel: CR2: 0000000000000000 CR3: 00000004c2610001 CR4: 00000000003706e0
Mai 30 20:41:07 ConceptD7 kernel: note: kworker/7:1[129] exited with irqs disabled

I hope this helps to narrow the cause.

Hmm, what kind of filesystem is used? There was some regression with XFS in 6.3.4.

You may want to write an email to linux-kernel@vger.kernel.org and linux-usb@vger.kernel.org with your DMESG RIP Log so the upstream kernel developers can figure it out. You can also test if the newer kernels fix your issue, as those are also built against stable branch.

So remember which exact last kernel in your series worked, which of that may broke your system and if a newer kernel maybe fixed it already. Also post which filessystem you’re using and if it is RAID or LVM.

Filesystem is ext4 and XFS for home. I can write an email and hope it will get fixed sooner or later. In the meantime I can stay with 5.x.
The exact last version that worked was 6.1.29 (currently on the live-ISO) while 6.1.30 did not. I don’t know for 6.2 or 6.3

Are the logs from booting kernel 6.x really wiped, or did your journalctl -b 0 only pick up the current boot for kernel 5.x?

See if you can find the logs for booting 6.x by different previous boot indexes, for example:

journalctl -b -2

Also, is your root partition encrypted?

Also, inxi reports no swap found. Which raises the question, could lack of swap be a root cause?

A new set of kernel updates got pushed. Changelog for 6.1.31 and others can be found here: kernel.org. Simply change the URL to the version you want to know what changed.

To figure out which version broke it, you may redownload the kernels from our github pages. URLs are also logical to been easily changed. Also check the pacman.log to give you clues what you updated from what. Plus you can check in our gitlab instance for changes we made to the kernels on top of it.

With a solid information with:

  • last working kernel
  • first broken kernel
  • crash report from DMESG logs
  • info about the used filesystem

… you may send an email to the kernel mailing lists to get help or notify the upstream kernel developers.

Thanks for the info about the kernel updates. 6.1.31 still hangs on login.
I did not knot you can filter journalctl with -b. The logs I posted is from ab boot with 6.1.30.

Looks like my issue got fixed with today update. I still don’t know what was the problem.
Should I close this topic or leave it open for others?

It depends if you (@pantas.aspro) had a similar problem like @mrcl has. You didn’t mentioned much why your system had an issue. Seems 6.1.30 introduced some issue on your end and the latest 6.1.31 fixed it for you.

So if there are kernel issues always provide as much info as possible. DMESG logs, hwinfo output and which packages got updated and the last working kernel of given series. You may also check the changelogs of the kernels to see what changed and what might possible affect your system.

Then others might be able to help better and similar affected people might find the ongoing discussion of the problem easier.

Sometimes it is drivers updates like from AMD …

It really looked like we have the same problem, as the error in the logs is the same.
Glad the latest kernel update solved your problem.
I think this topic can be closed anyway. As philm wrote, I better write to the kernel developers.

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.