NVIDIA driver fails to load after latest updates

Support Graphics & Display
,
1

Hi there,

Currently I use the ‘video-nvidia’ graphics driver on my Manjaro system. This morning after installing a number of available updates using the Package Manager GUI app, my system hung while booting.

On investigation journalctl showed that it seems to be a mismatch between the latest kernel and the current NVIDIA kernel module version.

The following lines from my system logs stood out:

Jun 05 11:43:48 _hostname_ systemd-modules-load[495]: Failed to insert module 'nvidia': Exec format error
Jun 05 11:43:48 _hostname_ kernel: loop2: detected capacity change from 0 to 315544
Jun 05 11:43:48 _hostname_ kernel: nvidia: version magic '5.15.114-1-MANJARO SMP preempt mod_unload ' should be '5.15.114-2-MANJARO SMP preempt mod_unload '
...
Jun 05 11:43:49 _hostname_ kernel: nvidia: version magic '5.15.114-1-MANJARO SMP preempt mod_unload ' should be '5.15.114-2-MANJARO SMP preempt mod_unload '
...
Jun 05 11:43:49 _hostname_ kernel: nvidia: version magic '5.15.114-1-MANJARO SMP preempt mod_unload ' should be '5.15.114-2-MANJARO SMP preempt mod_unload '
Jun 05 11:43:49 _hostname_ systemd-modules-load[495]: Failed to insert module 'nvidia_drm': Exec format error
...
Jun 05 11:43:50 _hostname_ kernel: nvidia: version magic '5.15.114-1-MANJARO SMP preempt mod_unload ' should be '5.15.114-2-MANJARO SMP preempt mod_unload '
...
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]: X.Org X Server 1.21.1.8
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]: X Protocol Version 11, Revision 0
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]: Current Operating System: Linux _hostname_ 5.15.114-2-MANJARO #1 SMP PREEMPT Sun Jun 4 10:32:43 UTC 2023 x86_64
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]: Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.15-x86_64 root=UUID=ff88a440-bf20-46cd-9fea-b40c60bca86d rw quiet splash apparmor=1 security=apparmor udev.log_priority=3
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]:  
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]: Current version of pixman: 0.42.2
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]:         Before reporting problems, check http://wiki.x.org
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]:         to make sure that you have the latest version.
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]: Markers: (--) probed, (**) from config file, (==) default setting,
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]:         (++) from command line, (!!) notice, (II) informational,
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]:         (WW) warning, (EE) error, (NI) not implemented, (??) unknown.
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]: (==) Log file: "/var/log/Xorg.0.log", Time: Mon Jun  5 11:44:01 2023
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]: (==) Using config directory: "/etc/X11/xorg.conf.d"
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]: (==) Using system config directory "/usr/share/X11/xorg.conf.d"
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]: (==) ServerLayout "layout"
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]: (**) |-->Screen "Screen0" (0)
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]: (**) |   |-->Monitor "Monitor0"
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]: (**) |   |-->Device "Device0"
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]: (==) Automatically adding devices
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]: (==) Automatically enabling devices
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]: (==) Automatically adding GPU devices
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]: (==) Automatically binding GPU devices
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]: (==) Max clients allowed: 256, resource mask: 0x1fffff
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]: (WW) The directory "/usr/share/fonts/misc" does not exist.
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]:         Entry deleted from font path.
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]: (WW) The directory "/usr/share/fonts/OTF" does not exist.
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]:         Entry deleted from font path.
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]: (WW) The directory "/usr/share/fonts/Type1" does not exist.
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]:         Entry deleted from font path.
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]: (WW) The directory "/usr/share/fonts/100dpi" does not exist.
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]:         Entry deleted from font path.
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]: (WW) The directory "/usr/share/fonts/75dpi" does not exist.
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]:         Entry deleted from font path.
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]: (==) FontPath set to:
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]:         /usr/share/fonts/TTF
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]: (==) ModulePath set to "/usr/lib/xorg/modules"
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]: (**) Extension "COMPOSITE" is enabled
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]: (II) The server relies on udev to provide the list of input devices.
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]:         If no devices become available, reconfigure udev or disable AutoAddDevices.
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]: (II) Module ABI versions:
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]:         X.Org ANSI C Emulation: 0.4
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]:         X.Org Video Driver: 25.2
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]:         X.Org XInput driver : 24.4
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]:         X.Org Server Extension : 10.0
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]: (++) using VT number 1
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]: (II) systemd-logind: took control of session /org/freedesktop/login1/session/_31
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]: (--) PCI:*(10@0:0:0) 10de:1e84:1043:8728 rev 161, Mem @ 0xfb000000/16777216, 0xffe0000000/268435456, 0xfff0000000/33554432, I/O @ 0x0000e000/128, BIOS @ 0x????????/524288
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]: (WW) Open ACPI failed (/var/run/acpid.socket) (No such file or directory)
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]: (II) LoadModule: "glx"
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]: (II) Loading /usr/lib/xorg/modules/extensions/libglx.so
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]: (II) Module glx: vendor="X.Org Foundation"
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]:         compiled for 1.21.1.8, module version = 1.0.0
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]:         ABI class: X.Org Server Extension, version 10.0
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]: (II) LoadModule: "nvidia"
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]: (II) Loading /usr/lib/xorg/modules/drivers/nvidia_drv.so
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]: (II) Module nvidia: vendor="NVIDIA Corporation"
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]:         compiled for 1.6.99.901, module version = 1.0.0
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]:         Module class: X.Org Video Driver
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]: (II) NVIDIA dlloader X Driver  530.41.03  Thu Mar 16 19:25:45 UTC 2023
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]: (II) NVIDIA Unified Driver for all Supported NVIDIA GPUs
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]: (II) Loading sub module "fb"
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]: (II) LoadModule: "fb"
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]: (II) Module "fb" already built-in
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]: (II) Loading sub module "wfb"
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]: (II) LoadModule: "wfb"
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]: (II) Loading /usr/lib/xorg/modules/libwfb.so
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]: (II) Module wfb: vendor="X.Org Foundation"
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]:         compiled for 1.21.1.8, module version = 1.0.0
Jun 05 11:44:01 _hostname_ /usr/lib/gdm-x-session[2126]:         ABI class: X.Org ANSI C Emulation, version 0.4
Jun 05 11:44:02 _hostname_ kernel: nvidia: version magic '5.15.114-1-MANJARO SMP preempt mod_unload ' should be '5.15.114-2-MANJARO SMP preempt mod_unload '
Jun 05 11:44:02 _hostname_ /usr/lib/gdm-x-session[2126]: (EE) NVIDIA: Failed to initialize the NVIDIA kernel module. Please see the
Jun 05 11:44:02 _hostname_ /usr/lib/gdm-x-session[2126]: (EE) NVIDIA:     system's kernel log for additional error messages and
Jun 05 11:44:02 _hostname_ /usr/lib/gdm-x-session[2126]: (EE) NVIDIA:     consult the NVIDIA README for details.
Jun 05 11:44:02 _hostname_ /usr/lib/gdm-x-session[2126]: (EE) No devices detected.
Jun 05 11:44:02 _hostname_ /usr/lib/gdm-x-session[2126]: (EE)
Jun 05 11:44:02 _hostname_ /usr/lib/gdm-x-session[2126]: Fatal server error:
Jun 05 11:44:02 _hostname_ /usr/lib/gdm-x-session[2126]: (EE) no screens found(EE)
Jun 05 11:44:02 _hostname_ /usr/lib/gdm-x-session[2126]: (EE)
Jun 05 11:44:02 _hostname_ /usr/lib/gdm-x-session[2126]: Please consult the The X.Org Foundation support
Jun 05 11:44:02 _hostname_ /usr/lib/gdm-x-session[2126]:          at http://wiki.x.org
Jun 05 11:44:02 _hostname_ /usr/lib/gdm-x-session[2126]:  for help.
Jun 05 11:44:02 _hostname_ /usr/lib/gdm-x-session[2126]: (EE) Please also check the log file at "/var/log/Xorg.0.log" for additional information.
Jun 05 11:44:02 _hostname_ /usr/lib/gdm-x-session[2126]: (EE)
Jun 05 11:44:02 _hostname_ /usr/lib/gdm-x-session[2126]: (EE) Server terminated with error (1). Closing log file.
Jun 05 11:44:02 _hostname_ /usr/lib/gdm-x-session[2124]: Unable to run X server
Jun 05 11:44:02 _hostname_ gdm[1826]: Gdm: GdmDisplay: Session never registered, failing

I was able to work-around the issue by switching to a console (Ctrl + Alt + F2), logging in, uninstalling the video-nvidia driver and installing the video-linux driver.

sudo mhwd -r pci video-nvidia
sudo mhwd -i pci video-linux

After confirming the desktop works with the video-linux driver, I re-installed the NVIDIA driver via the Manjaro Settings Manager GUI app just to see if it would solve the issue, however the system still hung without starting the Xserver, so I once again reverted and it was fine.

Here is my system information:

  • Kernel: 5.15.114-2-MANJARO arch: x86_64 bits: 64 compiler: gcc v: 13.1.1
  • Desktop: GNOME v: 44.1
  • Distro: Manjaro Linux base: Arch Linux
  • Mobo: Gigabyte model: X570 AORUS PRO WIFI
  • UEFI: American Megatrends LLC. v: F35d date: 10/13/2021
  • CPU: 12-core model: AMD Ryzen 9 3900X
  • Graphics: Device-1: NVIDIA TU104 [GeForce RTX 2070 SUPER] vendor: ASUSTeK

Hope this is enough information, and I welcome suggestions if anyone has any information about anything that I can do to resolve the issue.

Thanks

2

Correct, that is the issue.

You have 5.15.114-2-MANJARO installed, but the nvidia module was precompiled against 5.15.114-1-MANJARO.

Looks to me like a failure of the maintainer, just checked right now with mbn:

Branch         : unstable
Name           : linux515
Version        : 5.15.114-2
Repository     : core
Build Date     : Sun 04 Jun 2023 12:31:15 
Packager       : Manjaro Build Server <build@manjaro.org>
Branch         : testing
Name           : linux515
Version        : 5.15.114-1
Repository     : core
Build Date     : Tue 30 May 2023 23:18:58 
Packager       : Manjaro Build Server <build@manjaro.org>
Branch         : stable
Name           : linux515
Version        : 5.15.114-2
Repository     : core
Build Date     : Sun 04 Jun 2023 12:31:15 
Packager       : Manjaro Build Server <build@manjaro.org>

Branch         : unstable
Name           : linux515-nvidia
Version        : 530.41.03-13
Repository     : extra
Build Date     : Sun 04 Jun 2023 15:14:11 
Packager       : Manjaro Build Server <build@manjaro.org>
Branch         : testing
Name           : linux515-nvidia
Version        : 530.41.03-12
Repository     : extra
Build Date     : Wed 31 May 2023 01:29:01 
Packager       : Manjaro Build Server <build@manjaro.org>
Branch         : stable
Name           : linux515-nvidia
Version        : 530.41.03-12
Repository     : extra
Build Date     : Wed 31 May 2023 01:29:01 
Packager       : Manjaro Build Server <build@manjaro.org>

So the kernel 5.15.114-2 just skipped the testing branch. xD

Maybe try downgrading the kernel to 5.15.114-1 if it is still in your cache for the time being.

@Yochanan @philm Please look into it. There is something wrong.

3

Thankyou for the reply, I swapped to the 6.1.31-2 LTS kernel and things are working with the video-nvidia driver again.

4

Yep, looks like it matches, although it seems to skip testing branch aswell:

Branch         : unstable
Name           : linux61
Version        : 6.1.31-2
Repository     : core
Build Date     : Sun 04 Jun 2023 12:33:38 
Packager       : Manjaro Build Server <build@manjaro.org>
Branch         : testing
Name           : linux61
Version        : 6.1.31-1
Repository     : core
Build Date     : Tue 30 May 2023 23:19:50 
Packager       : Manjaro Build Server <build@manjaro.org>
Branch         : stable
Name           : linux61
Version        : 6.1.31-2
Repository     : core
Build Date     : Sun 04 Jun 2023 12:33:38 
Packager       : Manjaro Build Server <build@manjaro.org>

Branch         : unstable
Name           : linux61-nvidia
Version        : 530.41.03-13
Repository     : extra
Build Date     : Sun 04 Jun 2023 14:59:43 
Packager       : Manjaro Build Server <build@manjaro.org>
Branch         : testing
Name           : linux61-nvidia
Version        : 530.41.03-12
Repository     : extra
Build Date     : Wed 31 May 2023 01:41:07 
Packager       : Manjaro Build Server <build@manjaro.org>
Branch         : stable
Name           : linux61-nvidia
Version        : 530.41.03-13
Repository     : extra
Build Date     : Sun 04 Jun 2023 14:59:43 
Packager       : Manjaro Build Server <build@manjaro.org>
5

There was a hiccup with the wrong kernels pushed to stable yesterday intially… the right kernels were pushed shortly after … I would think by now users could resync for the up to date ones.

6

What do you mean? How to resync? I’ve refreshed the mirrors, there are no updates. How to install the kernel that works? Is there a workaround?

7

I’ve ran into the same issue today. I’m a casual linux user but fortunately thanks to this forum I was able to piece together the knowledge to perform the downgrade to 5.15.114-1. I’ll be sure to postpone big updates by at least a week from now on. When can the correct module be expected to be available?

8

How did you downgrade?

With this rolling release thing it breaks every fifth upgrade and you don’t have the old packages to install.

9

Enter TTY with the CTRL+ALT+F2 shortcut first, log in.

Assuming you’re running the 5.15 kernel, search the cache for the previous versions with:

ls -hl /var/cache/pacman/pkg | grep linux515

If it’s on the list - install it with:

sudo pacman -U /var/cache/pacman/pkg/linux515-5.15.114-1-x86_64.pkg.tar.zst

Afterwards I also ran:

sudo update-grub
sudo mkinitcpio -P
reboot
3 Likes
10

I don’t have that cache file.

11

That’s a bummer. Hopefully someone more knowledgeable chimes in to help. You could try searching for information on how to install a specific kernel version in the meantime

12

Let me check if we had some build errors: Release 5.15.114-2 · manjaro-kernels/linux515 · GitHub
Seems extramodules got only partly pushed to stable and testing branches …

13

Hey dea!

I was able to work around it by removing the video-nvidia driver and swapping to the video-linux (Open Source) driver. At least that got the X server running again.

How I swapped to the Open Source driver:

  1. Boot and wait till the system hangs (give it enough time).
  2. Eventually you can press ALT + CTRL + F2 to get to the terminal.
  3. Log in with your username and password.
  4. Remove the video-nvidia driver.
sudo mhwd -r pci video-nvidia
  1. Install the video-linux driver.
sudo mhwd -i pci video-linux
  1. Next, reboot the PC
sudo shutdown -r now
  1. Your system should boot into the Wayland/Xserver.

This is only a work-around, you’ll be using the Open Source driver and not the proprietary NVIDIA one, but at least you’ll be on your desktop!

How I got the NVIDIA driver working again:

  1. Launch “Manjaro Settings Manager”.
  2. Select the “Kernel” option.
  3. Choose another kernel version. In my case I went from an older “5.15.114-2 LTS” version to “6.1.31-2 LTS” version. If you’re on the latest LTS, maybe you could try downgrading to an earlier LTS, or installing a non LTS version.
  4. After the kernel installs, reboot the system. If you downgraded, you may need to select the appropriate kernel version from GRUB on booting.
  5. When you’ve booted into the new kernel version, run “Manjaro Settings Manager” again.
  6. This time choose “hardware Configuration”.
  7. Right click on the video-nvidia option, and choose “Install”.
  8. Optional. If everything works on the new version of the kernel, you could uninstall the older version.

This worked for me, I think the reason it worked was because “Manjaro Settings Manager” compiled the kernel module for me rather than downloading a pre-built package.

Hopefully it might work for you, however, try at your own risk! Good luck!

2 Likes
14

Thank you! This worked perfect for me.

Installed latest updates this morning, after work loaded it up to…you know, not load it up haha

16

Manjaro Team,

Can your team get your act together and test everything before pushing the update to the public? It’s not very hard and Nvidia is a pretty standard driver and it happens on boot so it catchable. Test it. Thanks.

I recalled the patching process yesterday, it hanged and hanged, I was about to kill the upgrade process then it did something afterwards 10 mins later and finished, but it booted with this issue today. Test things, Manjaro.

17

Manjaro always has been a community focused distribution, we dont have the man power to test 16.000 packages or 100 machines at our disposal for hardware testing, that is why the update announcements are for, so the community can send feedback and do their own QA on their own hardware so we can provide fixes, You can participate in testing in the community testing area.

3 Likes
18

I also found the pc stuck at boot after the update.
I temporarily solved it by removing the 5.15 kernel, now in fact I’m on the old 5.10. When will I be able to reinstall the 5.15 kernel without locking the machine again?

19

I’ve been using nvidia-dkms from the repo for several years now with no issue’s.Whenever there’s a kernel update it rebuilds the driver for the kernel.The only downside I’ve found is when I install a newer version kernel like from 6.2 to 6.3 I have to open pamac click on nvidia-dkms and reinstall.That builds the nvidia driver for the newer kernel then I can reboot to the newer kernel.If your running an LTS kernel you wouldn’t have to worry about reinstalling it for years.I can’t remember who on the forum suggested it but it’s worked out very well.

1 Like
20

The reason why reinstalling it works is because it triggers the mkinitcpio hook. Use the following hook instead:

/etc/pacman.d/hooks/90-mkinitcpio-dkms-linux.hook:

[Trigger]
Operation=Install
Operation=Upgrade
Operation=Remove
Type=Package
Target=linux
Target=*dkms

[Action]
Description=Update dkms modules in Linux initcpio
Depends=mkinitcpio
When=PostTransaction
NeedsTargets
Exec=/usr/bin/mkinitcpio -P
21

Thank you very much for that.