Failed to start kernel modules

I was using an unstable kernel for some reason and had to remove it because it became unsupported. However, I can now no longer boot into my system, it goes straight into emergency mode. I suspect this is an nVida driver issue from the xjournal entries, but I am not sure. I have tried:

  • Using manjaro-chroot on a USB to run a full system upgrade and update grub. This does not seem to help. update-grub returns:
Generating grub configuration file ...
Found theme: /usr/share/grub/themes/manjaro/theme.txt
Found linux image: /boot/vmlinuz-5.13-x86_64
Found initrd image: /boot/intel-ucode.img /boot/initramfs-5.13-x86_64.img
Found initrd fallback image: /boot/initramfs-5.13-x86_64-fallback.img
Found linux image: /boot/vmlinuz-5.10-x86_64
Found initrd image: /boot/intel-ucode.img /boot/initramfs-5.10-x86_64.img
Found initrd fallback image: /boot/initramfs-5.10-x86_64-fallback.img
Found linux image: /boot/vmlinuz-5.9-x86_64
Found initrd image: /boot/intel-ucode.img /boot/initramfs-5.9-x86_64.img
Found initrd fallback image: /boot/initramfs-5.9-x86_64-fallback.img
Found linux image: /boot/vmlinuz-5.4-x86_64
Found initrd image: /boot/intel-ucode.img /boot/initramfs-5.4-x86_64.img
Found initrd fallback image: /boot/initramfs-5.4-x86_64-fallback.img
Warning: os-prober will be executed to detect other bootable partitions.
Its output will be used to detect bootable binaries on them and create new boot entries.
grub-probe: error: cannot find a GRUB drive for /dev/sdc1.  Check your device.map.
grub-probe: error: cannot find a GRUB drive for /dev/sdc1.  Check your device.map.
Adding boot menu entry for UEFI Firmware Settings ...
Found memtest86+ image: /boot/memtest86+/memtest.bin
done
  • Adding a new kernel using pacman. I currently have
[manjaro /]# mhwd-kernel -li
Currently running: 5.9.16-1-MANJARO (linux59)
The following kernels are installed in your system:
   * linux510
   * linux54

But none of thoe show up in grub. I am unsure how to remove kernels - updating through mhwd does not work (no targets specified error when I def. have a valid kernel) so I can use the rmc option.

  • Reinstalling nvida drivers. Current output of mhwd is:
[manjaro /]# mhwd -li
> Installed PCI configs:
--------------------------------------------------------------------------------
                  NAME               VERSION          FREEDRIVER           TYPE
--------------------------------------------------------------------------------
          video-nvidia            2021.12.18               false            PCI
           video-linux            2018.05.04                true            PCI


Warning: No installed USB configs!
  • update-grub && mkinitcpio -p and ‘grub-mkconfig’ which both do not help

That is all the advice for this issue I have been able to find on the forum or could think of with my own knowledge.

Addition info:

What the system boots to: https://imgur.com/a/5yf0Tk9

Relevant-looking section of journalctl: https://imgur.com/a/EkMtLwA

When entering manjaro-chroot I get the error message

grub-probe: error: cannot fin a GRUB drive or /dev/sdc1. CHeck your devices.map.
grub-probe: error: cannot find a GRUB drive for /dev/sdc1. Check your devices.map.
==> mounting (ManjaroLinux) [/dev/sda2]
    --> mount [/mnt]
    --> mount [/mnt/boot/efi]

Are you chrooting in /dev/sda or /dev/sdc ?

Manjaro Chroot

Also this might help
Restore Grub Bootloader

I am chrooting into /dev/sda2 which seems to be correct from a look at partitions. /dev/sdc is the USB itself, so I think this error is probably a red herring. For reference:

/dev/sda: My boot SSD
/dev/sdb: My storage drive
/dev/sdc: The recovery USB

I have followed the grub restoration with no change.

[manjaro /]# pacman -Syu grub
:: Synchronizing package databases...
 core is up to date
 extra is up to date
 community is up to date
 multilib is up to date
warning: grub-2.06.r261.g2f4430cc0-1 is up to date -- reinstalling
:: Starting full system upgrade...
resolving dependencies...
looking for conflicting packages...

Packages (1) grub-2.06.r261.g2f4430cc0-1

Total Installed Size:  46.79 MiB
Net Upgrade Size:       0.00 MiB

:: Proceed with installation? [Y/n] y
(1/1) checking keys in keyring                     [##################] 100%
(1/1) checking package integrity                   [##################] 100%
(1/1) loading package files                        [##################] 100%
(1/1) checking for file conflicts                  [##################] 100%
(1/1) checking available disk space                [##################] 100%
:: Processing package changes...
(1/1) reinstalling grub                            [##################] 100%
:: Grub does no longer support side-loading modules when secure boot is
   enabled. Thus booting will fail, unless you have an efi executable
   'grubx64.efi' with bundled modules.
:: Running post-transaction hooks...
(1/2) Arming ConditionNeedsUpdate...
(2/2) Updating the info directory file...
[manjaro /]# grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=manjaro --recheck
Installing for x86_64-efi platform.
Installation finished. No error reported.
[manjaro /]# grub-mkconfig -o /boot/grub/grub.cfg
Generating grub configuration file ...
Found theme: /usr/share/grub/themes/manjaro/theme.txt
Found linux image: /boot/vmlinuz-5.13-x86_64
Found initrd image: /boot/intel-ucode.img /boot/initramfs-5.13-x86_64.img
Found initrd fallback image: /boot/initramfs-5.13-x86_64-fallback.img
Found linux image: /boot/vmlinuz-5.10-x86_64
Found initrd image: /boot/intel-ucode.img /boot/initramfs-5.10-x86_64.img
Found initrd fallback image: /boot/initramfs-5.10-x86_64-fallback.img
Found linux image: /boot/vmlinuz-5.9-x86_64
Found initrd image: /boot/intel-ucode.img /boot/initramfs-5.9-x86_64.img
Found initrd fallback image: /boot/initramfs-5.9-x86_64-fallback.img
Found linux image: /boot/vmlinuz-5.4-x86_64
Found initrd image: /boot/intel-ucode.img /boot/initramfs-5.4-x86_64.img
Found initrd fallback image: /boot/initramfs-5.4-x86_64-fallback.img
Warning: os-prober will be executed to detect other bootable partitions.
Its output will be used to detect bootable binaries on them and create new boot entries.
grub-probe: error: cannot find a GRUB drive for /dev/sdc1.  Check your device.map.
grub-probe: error: cannot find a GRUB drive for /dev/sdc1.  Check your device.map.
Adding boot menu entry for UEFI Firmware Settings ...
Found memtest86+ image: /boot/memtest86+/memtest.bin
done

you have there 2 eol kernels - 5.13 and 5.9, post output from:
mhwd-kernel -li
ls /etc/mkinitcpio.d/*.preset
ls /boot/initramfs-*
ls /boot/vmlinuz-*
ls /usr/lib/modules/*
also how long did you not update your system?

Not that long, I had one out of date kernel (5.9) which I removed a few days after it became EOL, which is what caused this issue. I think I must have installed 5.13 while trying to fix it. Or I didn’t remove it properly, I’ve always dealt with kernels through the manjaro GUI out of laziness and the hope that that was safer. But I def. only 5.9 installed when there was a problem (which was dumb) and tried to replace it with the two most recent LTS ones.

output from this

  • mhwd-kernel -li
[manjaro /]# mhwd-kernel -li
Currently running: 5.9.16-1-MANJARO (linux59)
The following kernels are installed in your system:
   * linux510
   * linux54
  • ls /etc/mkinitcpio.d/*.preset
[manjaro /]#  ls /etc/mkinitcpio.d/*.preset
/etc/mkinitcpio.d/linux510.preset  /etc/mkinitcpio.d/linux54.preset
/etc/mkinitcpio.d/linux513.preset  /etc/mkinitcpio.d/linux59.preset

  • ls /boot/initramfs-*
/boot/initramfs-5.10-x86_64-fallback.img
/boot/initramfs-5.10-x86_64.img
/boot/initramfs-5.13-x86_64-fallback.img
/boot/initramfs-5.13-x86_64.img
/boot/initramfs-5.4-x86_64-fallback.img
/boot/initramfs-5.4-x86_64.img
/boot/initramfs-5.9-x86_64-fallback.img
/boot/initramfs-5.9-x86_64.img
  • ls /boot/vmlinuz-*
/boot/vmlinuz-5.10-x86_64  /boot/vmlinuz-5.4-x86_64
/boot/vmlinuz-5.13-x86_64  /boot/vmlinuz-5.9-x86_64
  • ls /usr/lib/modules/*
/usr/lib/modules/5.10.124-1-MANJARO:
extramodules  kernel  kernelbase  modules.alias  modules.alias.bin  modules.builtin  modules.builtin.alias.bin  modules.builtin.bin  modules.builtin.modinfo  modules.dep  modules.dep.bin  modules.devname  modules.order  modules.softdep  modules.symbols  modules.symbols.bin  pkgbase  vmlinuz

/usr/lib/modules/5.4.200-1-MANJARO:
extramodules  kernel  kernelbase  modules.alias  modules.alias.bin  modules.builtin  modules.builtin.alias.bin  modules.builtin.bin  modules.builtin.modinfo  modules.dep  modules.dep.bin  modules.devname  modules.order  modules.softdep  modules.symbols  modules.symbols.bin  pkgbase  vmlinuz

/usr/lib/modules/extramodules-5.10-MANJARO:
nvidia-drm.ko.xz  nvidia-modeset.ko.xz  nvidia-peermem.ko.xz  nvidia-uvm.ko.xz  nvidia.ko.xz  version

/usr/lib/modules/extramodules-5.4-MANJARO:
nvidia-drm.ko.xz  nvidia-modeset.ko.xz  nvidia-peermem.ko.xz  nvidia-uvm.ko.xz  nvidia.ko.xz  version


so remove these:
rm /etc/mkinitcpio.d/linux513.preset
rm /etc/mkinitcpio.d/linux59.preset
rm /boot/initramfs-5.13-x86_64-fallback.img
rm /boot/initramfs-5.13-x86_64.img
rm /boot/initramfs-5.9-x86_64-fallback.img
rm /boot/initramfs-5.9-x86_64.img
rm /boot/vmlinuz-5.13-x86_64
rm /boot/vmlinuz-5.9-x86_64
then post output from:
inxi -xG
mhwd -l && mhwd -li

[manjaro /]#  inxi -xG
12Graphics:
  12Device-1 NVIDIA GP106 [GeForce GTX 1060 6GB] 12vendor ASUSTeK 12driver nvidia 12v 455.45.01
    12arch Pascal 12bus-ID 01:00.0
  12Display 12server X.Org 12v 1.20.10 12driver 12X 12loaded nvidia 12gpu nvidia 12resolution 2560x1440~60Hz
  12OpenGL 12renderer N/A 12v N/A 12direct render N/A
[manjaro /]#  mhwd -l && mhwd -li
> 0000:01:00.0 (0300:10de:1c03) Display controller nVidia Corporation:
--------------------------------------------------------------------------------
                  NAME               VERSION          FREEDRIVER           TYPE
--------------------------------------------------------------------------------
          video-nvidia            2021.12.18               false            PCI
    video-nvidia-470xx            2021.12.18               false            PCI
    video-nvidia-390xx            2021.12.18               false            PCI
           video-linux            2018.05.04                true            PCI
     video-modesetting            2020.01.13                true            PCI
            video-vesa            2017.03.12                true            PCI


> 0000:05:00.0 (0200:10ec:8168) Network controller Realtek Semiconductor Co., Ltd.:
--------------------------------------------------------------------------------
                  NAME               VERSION          FREEDRIVER           TYPE
--------------------------------------------------------------------------------
         network-r8168            2016.04.20                true            PCI


> Installed PCI configs:
--------------------------------------------------------------------------------
                  NAME               VERSION          FREEDRIVER           TYPE
--------------------------------------------------------------------------------
          video-nvidia            2021.12.18               false            PCI
           video-linux            2018.05.04                true            PCI


Warning: No installed USB configs!
[manjaro /]# 

reinstall your kernels:
pacman -S linux54 linux510
install the 515:
pacman -S linux515
reinstall drivers:
mhwd -r pci video-linux
mhwd -r pci video-nvidia
install:
mhwd -i pci video-nvidia
exit
reboot and in the grub menu select the 515 kernel

Thank you so much, that worked!

I don’t suppose there’s any chance you could explain what went wrong/point me to some relevant resources so I can fix it myself if it ever happens again?

very likely it was because of the eol kernels, and maybe something got messed up when you tried to remove them…

That makes sense, thank you.

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.