NVIDIA NVML: Driver/library version mismatch

nmcbride · 25 December 2022 22:31

Hello everyone,

I am having an issue with the nvidia driver. Technically the nvidia-open driver installed from the Manjaro git repo. I do wish this would be added into one of the official repos soon. I thought I saw something about it being in testing, but I switched and did a search and it doesn’t seem to be in there. Anywho…

I just updated the driver from 520.56.06 to 525.60.11 but can’t for the life of me get to get it to work. I am getting the following error:

$ cat /proc/driver/nvidia/version                              255 ✘ 
NVRM version: NVIDIA UNIX Open Kernel Module for x86_64  520.56.06  Release Build  (manjarolinux-builder@framework)  
GCC version:  gcc version 12.2.0 (GCC)

And in dmesg:

$ sudo dmesg | grep NVRM                                           ✔ 
[ 5542.653505] NVRM: loading NVIDIA UNIX Open Kernel Module for x86_64  520.56.06  Release Build  (manjarolinux-builder@framework)  
[ 5553.286873] NVRM: API mismatch: the client has the version 525.60.11, but
               NVRM: this kernel module has the version 520.56.06.  Please
               NVRM: make sure that this kernel module and all NVIDIA driver
               NVRM: components have the same version.

Researching, it seems like the issue is the packages updated, however the driver in the kernel is still the older version. I have searched a ton and tried all the fixes I could find but none of them have worked for me:

Rebooting
- I was hoping it would be this simple but this didn’t change anything for me.
Running mkinicpio -P
- I was really hopeful this would fix the issue. Rebuilding the kernel modules makes sense. The package is dkms though and rebuilds when I update it. However, forcing a rebuild didn’t change anything.
Reinstalling the packages
- I didn’t think this would help but tried it anyway to no avail.

Any help would be greatly appreciated. It really seems like the kernel module isn’t getting updated but I’m not sure why or what to do.

Thank you.

megavolt · 26 December 2022 05:00

Check the packages:

 pamac list --installed | grep nvidia

Is there a version mismatch?

Switch global mirror and do an update:

sudo pacman-mirrors -c Global
pamac update --force-refresh --enable-downgrade

nmcbride · 26 December 2022 05:11

The packages look all correct.

From what I’ve read on other threads this is specifically related to a kernel module vs packages version.

Here are the packages:

$ sudo pacman -Qs nvidia                                                                               
...
local/libxnvctrl 525.60.11-4
    NVIDIA NV-CONTROL X extension
local/mhwd-nvidia 525.60.11-2
    MHWD module-ids for nvidia 525.60.11
local/nvidia-open-dkms 525.60.11-2
    NVIDIA open kernel modules
local/nvidia-prime 1.0-4
    NVIDIA Prime Render Offload configuration and utilities
local/nvidia-utils 525.60.11-2
    NVIDIA drivers utilities
local/opencl-nvidia 525.60.11-2
    OpenCL implemention for NVIDIA

And the version of what is loaded into the kernel even after a mkinitcpio -P that I showed in my first post.

$ cat /proc/driver/nvidia/version                              255 ✘ 
NVRM version: NVIDIA UNIX Open Kernel Module for x86_64  520.56.06  Release Build  (manjarolinux-builder@framework)  
GCC version:  gcc version 12.2.0 (GCC)

So the error is absolute correct:

$ sudo dmesg | grep NVRM                                           ✔ 
[ 5542.653505] NVRM: loading NVIDIA UNIX Open Kernel Module for x86_64  520.56.06  Release Build  (manjarolinux-builder@framework)  
[ 5553.286873] NVRM: API mismatch: the client has the version 525.60.11, but
               NVRM: this kernel module has the version 520.56.06.  Please
               NVRM: make sure that this kernel module and all NVIDIA driver
               NVRM: components have the same version.

But for some reason nothing I try is actually updating the kernel module.

megavolt · 26 December 2022 05:44

Well, I cannot find this module in the official repo, so I assume it is AUR. The normal package is nvidia-dkms or just precompiled linux[version]-nvidia.

You need to modify the PKGBUILD then to get most recent module version.

nmcbride · 26 December 2022 06:47

It’s from Manjaro’s gitlab directly as it is not (to my knowledge) pulled into a repo yet, but it’s not from AUR.

Here is the link:

But it should be up-to-date as it’s from the official Manjaro team and they just updated it three weeks ago which is why I’m in this mess.

If I install it on a clean Manjaro, the correct module is built. But upgrading my current system, it doesn’t appear the module is being rebuilt on my system.

nmcbride · 27 December 2022 05:24

Other than rolling back to the previous version that for some reason will not leave the kernel I’m not sure what else to do.

megavolt · 27 December 2022 07:22

If you are not sure what has to be done and how to recompile the dkms module, then I guess you are better off using the provided nvidia-dkms instead of nvidia-open-dkms which is not ready yet even in unstable branch. So please keep in mind that you are actually a tester here.

When I am at home again, sure I could test it, but right now, I can only say:

Fetch/Pull the source again.
Check the called commit or replace it.
Create a new package and install it.

nmcbride · 27 December 2022 18:50

I’ve never had issues recompiling other DKMS modules and I did recompile these multiple times and ran mkinitcpio -P every time after as well. This isn’t the first time I’ve had to to do this I’ve updated this multiple times. It is however the first time I’ve had this issue.

That’s basically the process yes with the only difference being you also have to update/build/install the nvidia-utils package to go along with it.

Unfortunately I’ve never been able to get the non-open drivers to work with my 3090 on Manjaro. But the open drivers have worked flawlessly until this update.

I’m happy to try any recommendation from anyone though to troubleshoot and help test / find a solution. Otherwise I guess I’ll either have to roll back to the previous version or switch to Arch for the time being since these are in the official repos there. I’d prefer not to though.

@Yochanan if you happen to have a moment would you mind taking a look over this thread and letting me know where I’m going wrong? Thanks very much!