Downgrade Nvidia drivers to 545

I’m having issues with Nvidia drivers causing kernel panic mostly while rebooting/shutdown and sometimes when starting up. The issues that I have match with the post on the Arch forums [SOLVED] Repeated kernel problems/freezes since 6.7.6 / Kernel & Hardware / Arch Linux Forums and 2 of the possible solutions listed there are:

  1. Blacklist nvidia_uvm
  2. Downgrade to 545

I’ve tried blacklisting nvidia_uvm and while this has improved the situation slightly, I still get kernel panic and random freezes. The issue with kernel panic is that I loose the logs (sysrq does not work) and log entries and file system gets corrupted. Worse, when the panic happens during shutdown/reboot, I can only see the ending of traceback which lists modules loaded and the final

kernel: note: systemd[993] exited with irqs disabled

It was pure luck that I managed to grab the logs during a startup and had the terminal open which led me to figure out it’s an Nvidia issue.

I searched through the forums and a general consensus seems to be that Nvidia drivers cannot be downgraded[0], however Downgrade nvidia 470.86 to 470.63 - #10 by philm from 3 years ago makes it seem like it’s possible. Is it still valid?

There is also another post Attempting to Downgrade Nvidia Driver; Unresolved Dependencies - #4 by cscs which points to mhwd but I can’t seem to figure out how this would allow me to go back to specific versions.

[0] I don't see an option to downgrade NVIDIA drivers - #3 by Sempo

I’m looking for 2 things:

  1. How do I downgrade (from 550 → 545 or earlier if this does not work out)?
  2. Once 555 is release restore things back the way it is currently i.e. move back to the latest branch (and any associated packages such as linux-nvidia*)

There are no nvidia-545 packages in the Manjaro or Arch repos, or even in the AUR.
This means that you cannot use mhwd or any downgrade tool to recover/switch to 545, nor can you simply install it.
If you happen to have certain packages in your cache then maybe … otherwise you will have to write/edit a PKGBUILD yourself.

If we are to assume you somehow managed to install nvidia-545 packages then you would re-perform regular mhwd selection. ex:

sudo mhwd -i pci video-nvidia

Though it could be likely that you will have dependencies to manage before that will go through.
In such an event you may have to remove some of the offending packages beforehand. You may also benefit from the force flag:

sudo mhwd -f- i pci video-nvidia

Hi @wpkg ,
I wonder whether you could try another kernel instead of linux67. Maybe, you could solve your problem.

In order to help you, this post will be useful to any other posts.
[HowTo] Request support

Hope it help, regards

It should also be considered that Linux 6.7.12 [EOL] so please change to supported kernel ASAP.

linux66 or linux68 is valid choices.

I managed to get 545 and will let it run for a week to see if it helps. philm’s instructions are clear and I had no issues building the packages. I needed to downgrade the following:

linux66-nvidia
linux61-nvidia
nvidia-utils
lib32-nvidia-utils

I picked the last 545 commit for these and built the packages. Also added these to ignore package in pacman.conf

I suspect a package could be missed here as the GUI app (it’s called Nvidia Server, I think) is not shown in application menu – pacman does not complain so whatever it is, is not a dependency. For now downgrading the driver, running the system normally and seeing if I continue to observe kernel panic are my next steps.

When the next version of drivers come out I can test it out by removing ignore package. I have btrfs + snapshots, so can go back and forth as needed.

It should also be considered that Linux 6.7.12 [EOL] so please change to supported kernel ASAP.
linux66 or linux68 is valid choices.

Noted. I’ve been using 6.6 and 6.1 LTS for the moment.

I wonder whether you could try another kernel instead of linux67. Maybe, you could solve your problem.

I’ve used 6.1, 6.6, 6.7 & 6.8 so far and have had issues with all of them. Frankly, downgrading Nvidia packages is in itself a test, but the symptoms plus the only journal I managed to capture match the Arch forum post.

In order to help you, this post will be useful to any other posts.
[HowTo] Request support

Here is what I have in the only journal I managed to retain, kernel panic (6.8) on VT and sysrq not working don’t help:

Apr 06 12:45:26 tufwrapper systemd[1]: Stopping User Manager for UID 964...
Apr 06 12:45:26 tufwrapper systemd[993]: Activating special unit Exit the Session...
Apr 06 12:45:26 tufwrapper systemd[993]: Stopped target Main User Target.
Apr 06 12:45:26 tufwrapper wireplumber[1103]: stopped by signal: Terminated
Apr 06 12:45:26 tufwrapper systemd[993]: Stopping Multimedia Service Session Manager...
Apr 06 12:45:26 tufwrapper systemd[993]: Stopping Portal service...
Apr 06 12:45:26 tufwrapper systemd[993]: Stopping flatpak document portal service...
Apr 06 12:45:26 tufwrapper systemd[993]: Stopping sandboxed app permission store...
Apr 06 12:45:26 tufwrapper systemd[993]: Stopped sandboxed app permission store.
Apr 06 12:45:26 tufwrapper wireplumber[1103]: disconnected from pipewire
Apr 06 12:45:26 tufwrapper wireplumber[1103]: 0x60516f728ff0: leaked proxy 0x60516f8ec180 id:18
Apr 06 12:45:26 tufwrapper wireplumber[1103]: 0x60516f728ff0: leaked proxy 0x60516f9229a0 id:19
Apr 06 12:45:26 tufwrapper wireplumber[1103]: 0x60516f728ff0: leaked proxy 0x60516f884d80 id:20
Apr 06 12:45:26 tufwrapper systemd[1]: run-user-964-doc.mount: Deactivated successfully.
Apr 06 12:45:26 tufwrapper systemd[993]: Stopped Portal service.
Apr 06 12:45:26 tufwrapper systemd[993]: Stopped flatpak document portal service.
Apr 06 12:45:26 tufwrapper systemd[993]: Stopped Multimedia Service Session Manager.
Apr 06 12:45:26 tufwrapper systemd[993]: Stopping PipeWire Multimedia Service...
Apr 06 12:45:26 tufwrapper systemd[993]: Stopped PipeWire Multimedia Service.
Apr 06 12:45:26 tufwrapper systemd[993]: Stopped target Basic System.
Apr 06 12:45:26 tufwrapper systemd[993]: Stopped target Paths.
Apr 06 12:45:26 tufwrapper systemd[993]: Stopped target Sockets.
Apr 06 12:45:26 tufwrapper systemd[993]: Stopped target Timers.
Apr 06 12:45:26 tufwrapper systemd[993]: Closed GnuPG network certificate management daemon.
Apr 06 12:45:26 tufwrapper systemd[993]: Closed GnuPG cryptographic agent and passphrase cache (access for web browsers).
Apr 06 12:45:26 tufwrapper systemd[993]: Closed GnuPG cryptographic agent and passphrase cache (restricted).
Apr 06 12:45:26 tufwrapper systemd[993]: Closed GnuPG cryptographic agent (ssh-agent emulation).
Apr 06 12:45:26 tufwrapper systemd[993]: Closed GnuPG cryptographic agent and passphrase cache.
Apr 06 12:45:26 tufwrapper systemd[993]: Closed GnuPG public key management service.
Apr 06 12:45:26 tufwrapper systemd[993]: Closed p11-kit server.
Apr 06 12:45:26 tufwrapper systemd[993]: Closed PipeWire PulseAudio.
Apr 06 12:45:26 tufwrapper systemd[993]: Closed PipeWire Multimedia System Sockets.
Apr 06 12:45:26 tufwrapper dbus-broker[1040]: Dispatched 819 messages @ 3(±5)μs / message.
Apr 06 12:45:26 tufwrapper systemd[993]: Stopping D-Bus User Message Bus...
Apr 06 12:45:26 tufwrapper systemd[993]: Stopped D-Bus User Message Bus.
Apr 06 12:45:26 tufwrapper systemd[993]: Removed slice User Core Session Slice.
Apr 06 12:45:26 tufwrapper kernel: BUG: kernel NULL pointer dereference, address: 00000000000000ca
Apr 06 12:45:26 tufwrapper kernel: #PF: supervisor read access in kernel mode
Apr 06 12:45:26 tufwrapper kernel: #PF: error_code(0x0000) - not-present page
Apr 06 12:45:26 tufwrapper kernel: PGD 0 P4D 0 
Apr 06 12:45:26 tufwrapper kernel: Oops: 0000 [#1] PREEMPT SMP NOPTI
Apr 06 12:45:26 tufwrapper kernel: CPU: 7 PID: 993 Comm: systemd Tainted: P           OE      6.8.4-1-MANJARO #1 58979b0609f62c42147855b1cae713a05bd8a64a
Apr 06 12:45:26 tufwrapper kernel: Hardware name: ASUSTeK COMPUTER INC. ASUS TUF Gaming F15 FX507ZE_FX577ZE/FX507ZE, BIOS FX507ZE.315 06/17/2022
Apr 06 12:45:26 tufwrapper kernel: RIP: 0010:rb_first+0xf/0x30
Apr 06 12:45:26 tufwrapper kernel: Code: 10 c3 cc cc cc cc 0f 1f 44 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 48 8b 07 48 85 c0 74 14 48 89 c2 <48> 8b 40 10 48 85 c0 75 f4 48 89 d0 c3 cc
 cc cc cc 31 d2 eb f4 66
Apr 06 12:45:26 tufwrapper kernel: RSP: 0018:ffffb89e41b67bd8 EFLAGS: 00010202
Apr 06 12:45:26 tufwrapper kernel: RAX: 00000000000000ba RBX: ffff9f687f551000 RCX: ffffe4a0040f1280
Apr 06 12:45:26 tufwrapper kernel: RDX: 00000000000000ba RSI: 0000000000000000 RDI: ffff9f68830846c8
Apr 06 12:45:26 tufwrapper kernel: RBP: ffff9f688179fc00 R08: 0000000000000246 R09: 00000000010000ff
Apr 06 12:45:26 tufwrapper kernel: R10: 00000000010000ff R11: 0000000000000002 R12: 0000000000000000
Apr 06 12:45:26 tufwrapper kernel: R13: ffff9f68830846c8 R14: ffff9f68826cd000 R15: ffffffff9450cf20
Apr 06 12:45:26 tufwrapper kernel: FS:  00007f13f9dbd640(0000) GS:ffff9f6be03c0000(0000) knlGS:0000000000000000
Apr 06 12:45:26 tufwrapper kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 06 12:45:26 tufwrapper kernel: CR2: 00000000000000ca CR3: 0000000131212000 CR4: 0000000000f50ef0
Apr 06 12:45:26 tufwrapper kernel: PKRU: 55555554
Apr 06 12:45:26 tufwrapper kernel: Call Trace:
Apr 06 12:45:26 tufwrapper kernel:  <TASK>
Apr 06 12:45:26 tufwrapper kernel:  ? __die+0x23/0x70
Apr 06 12:45:26 tufwrapper kernel:  ? page_fault_oops+0x171/0x4e0
Apr 06 12:45:26 tufwrapper kernel:  ? exc_page_fault+0x7f/0x180
Apr 06 12:45:26 tufwrapper kernel:  ? asm_exc_page_fault+0x26/0x30
Apr 06 12:45:26 tufwrapper kernel:  ? rb_first+0xf/0x30
Apr 06 12:45:26 tufwrapper kernel:  simple_xattrs_free+0x29/0x90
Apr 06 12:45:26 tufwrapper kernel:  kernfs_put.part.0+0x60/0x150
Apr 06 12:45:26 tufwrapper kernel:  kernfs_remove_by_name_ns+0x81/0xd0
Apr 06 12:45:26 tufwrapper kernel:  cgroup_addrm_files+0x2ca/0x340
Apr 06 12:45:26 tufwrapper kernel:  css_clear_dir+0x4b/0xc0
Apr 06 12:45:26 tufwrapper kernel:  cgroup_destroy_locked+0xcd/0x1b0
Apr 06 12:45:26 tufwrapper kernel:  cgroup_rmdir+0x2b/0xd0
Apr 06 12:45:26 tufwrapper kernel:  kernfs_iop_rmdir+0x50/0x80
Apr 06 12:45:26 tufwrapper kernel:  vfs_rmdir+0x97/0x200
Apr 06 12:45:26 tufwrapper kernel:  do_rmdir+0x1a9/0x1c0
Apr 06 12:45:26 tufwrapper kernel:  __x64_sys_rmdir+0x42/0x70
Apr 06 12:45:26 tufwrapper kernel:  do_syscall_64+0x86/0x170
Apr 06 12:45:26 tufwrapper kernel:  ? do_syscall_64+0x96/0x170
Apr 06 12:45:26 tufwrapper kernel:  ? syscall_exit_to_user_mode_prepare+0x178/0x1a0
Apr 06 12:45:26 tufwrapper kernel:  ? syscall_exit_to_user_mode+0x80/0x230
Apr 06 12:45:26 tufwrapper kernel:  ? do_syscall_64+0x96/0x170
Apr 06 12:45:26 tufwrapper kernel:  ? do_syscall_64+0x96/0x170
Apr 06 12:45:26 tufwrapper kernel:  ? syscall_exit_to_user_mode+0x80/0x230
Apr 06 12:45:26 tufwrapper kernel:  ? do_syscall_64+0x96/0x170
Apr 06 12:45:26 tufwrapper kernel:  ? do_syscall_64+0x96/0x170
Apr 06 12:45:26 tufwrapper kernel:  ? do_syscall_64+0x96/0x170
Apr 06 12:45:26 tufwrapper kernel:  ? do_syscall_64+0x96/0x170
Apr 06 12:45:26 tufwrapper kernel:  ? __irq_exit_rcu+0x4b/0xc0
Apr 06 12:45:26 tufwrapper kernel:  entry_SYSCALL_64_after_hwframe+0x73/0x7b
Apr 06 12:45:26 tufwrapper kernel: RIP: 0033:0x7f13fa71977b
Apr 06 12:45:26 tufwrapper kernel: Code: f0 ff ff 73 01 c3 48 8b 0d a2 c5 0d 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 54 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 05 c3 0f 1f 40 00 48 8b 15 71 c5 0d 00 f7 d8
Apr 06 12:45:26 tufwrapper kernel: RSP: 002b:00007ffd283e2b98 EFLAGS: 00000246 ORIG_RAX: 0000000000000054
Apr 06 12:45:26 tufwrapper kernel: RAX: ffffffffffffffda RBX: 00005c7e19972a10 RCX: 00007f13fa71977b
Apr 06 12:45:26 tufwrapper kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: 00005c7e19a0f2c0
Apr 06 12:45:26 tufwrapper kernel: RBP: 00007ffd283e2bf0 R08: 00007f13fa8c490a R09: 0000000000000007
Apr 06 12:45:26 tufwrapper kernel: R10: 00005c7e19a11260 R11: 0000000000000246 R12: 0000000000000001
Apr 06 12:45:26 tufwrapper kernel: R13: 0000000000000000 R14: 00005c7e19a0f2c0 R15: 0000000000000000
Apr 06 12:45:26 tufwrapper kernel:  </TASK>
Apr 06 12:45:26 tufwrapper kernel: Modules linked in: ccm xt_mark xt_connmark xt_comment nft_compat rfcomm snd_seq_dummy snd_hrtimer snd_seq udp_diag nf_conntrack_netlink nft_queue nfnetlink_queue tcp_diag inet_diag nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nf_log_syslog nft_log nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables qrtr cmac algif_hash algif_skcipher af_alg bnep snd_sof_pci_intel_tgl snd_sof_intel_hda_common soundwire_intel snd_sof_intel_hda_mlink soundwire_cadence snd_sof_intel_hda snd_sof_pci snd_sof_xtensa_dsp snd_sof tcp_bbr snd_sof_utils snd_soc_hdac_hda sch_cake snd_hda_ext_core snd_soc_acpi_intel_match snd_soc_acpi soundwire_generic_allocation soundwire_bus snd_soc_core snd_compress ac97_bus snd_pcm_dmaengine nvidia_uvm(POE) intel_uncore_frequency intel_uncore_frequency_common iwlmvm x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel snd_hda_codec_hdmi mac80211 kvm
Apr 06 12:45:26 tufwrapper kernel:  snd_hda_intel snd_intel_dspcfg libarc4 uvcvideo snd_intel_sdw_acpi snd_usb_audio btusb videobuf2_vmalloc snd_hda_codec processor_thermal_device_pci btrtl snd_usbmidi_lib irqbypass uvc btintel snd_ump snd_hda_core processor_thermal_device videobuf2_memops iwlwifi btbcm hid_multitouch iTCO_wdt processor_thermal_wt_hint mei_hdcp snd_rawmidi videobuf2_v4l2 processor_thermal_rfim snd_hwdep rapl btmtk asus_nb_wmi intel_pmc_bxt vfat snd_seq_device mei_pxp fat intel_rapl_msr iTCO_vendor_support videodev intel_cstate asus_wmi snd_pcm mei_me processor_thermal_rapl bluetooth e1000e cfg80211 spi_nor ledtrig_audio videobuf2_common intel_rapl_common ucsi_acpi intel_lpss_pci platform_profile snd_timer ecdh_generic i2c_i801 ptp intel_uncore wmi_bmof processor_thermal_wt_req pcspkr intel_lpss typec_ucsi mei mc mtd pps_core crc16 snd idma64 rfkill i2c_smbus typec processor_thermal_power_floor thunderbolt soundcore processor_thermal_mbox igen6_edac roles int3403_thermal intel_pmc_core int340x_thermal_zone nvidia_drm(POE)
Apr 06 12:45:26 tufwrapper kernel:  intel_vsec mousedev i2c_hid_acpi intel_hid int3400_thermal pmt_telemetry sparse_keymap acpi_pad i2c_hid joydev pinctrl_tigerlake acpi_thermal_rel pmt_class mac_hid nvidia_modeset(POE) nvidia(POE) i2c_dev loop crypto_user fuse nfnetlink ip_tables x_tables btrfs blake2b_generic libcrc32c crc32c_generic xor raid6_pq ses enclosure scsi_transport_sas uas usb_storage hid_generic usbhid dm_crypt cbc encrypted_keys trusted asn1_encoder tee dm_mod nvme nvme_core nvme_auth xe crct10dif_pclmul crc32_pclmul polyval_clmulni polyval_generic gf128mul ghash_clmulni_intel sha512_ssse3 sha256_ssse3 serio_raw drm_ttm_helper sha1_ssse3 atkbd gpu_sched aesni_intel libps2 drm_suballoc_helper vivaldi_fmap crypto_simd drm_gpuvm xhci_pci spi_intel_pci cryptd xhci_pci_renesas vmd drm_exec spi_intel i8042 serio i915 i2c_algo_bit drm_buddy video wmi ttm intel_gtt drm_display_helper cec crc32c_intel
Apr 06 12:45:26 tufwrapper kernel: CR2: 00000000000000ca
Apr 06 12:45:26 tufwrapper kernel: ---[ end trace 0000000000000000 ]---
Apr 06 12:45:26 tufwrapper kernel: RIP: 0010:rb_first+0xf/0x30
Apr 06 12:45:26 tufwrapper kernel: Code: 10 c3 cc cc cc cc 0f 1f 44 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 48 8b 07 48 85 c0 74 14 48 89 c2 <48> 8b 40 10 48 85 c0 75 f4 48 89 d0 c3 cc cc cc cc 31 d2 eb f4 66
Apr 06 12:45:26 tufwrapper kernel: RSP: 0018:ffffb89e41b67bd8 EFLAGS: 00010202
Apr 06 12:45:26 tufwrapper kernel: RAX: 00000000000000ba RBX: ffff9f687f551000 RCX: ffffe4a0040f1280
Apr 06 12:45:26 tufwrapper kernel: RDX: 00000000000000ba RSI: 0000000000000000 RDI: ffff9f68830846c8
Apr 06 12:45:26 tufwrapper kernel: RBP: ffff9f688179fc00 R08: 0000000000000246 R09: 00000000010000ff
Apr 06 12:45:26 tufwrapper kernel: R10: 00000000010000ff R11: 0000000000000002 R12: 0000000000000000
Apr 06 12:45:26 tufwrapper kernel: R13: ffff9f68830846c8 R14: ffff9f68826cd000 R15: ffffffff9450cf20
Apr 06 12:45:26 tufwrapper kernel: FS:  00007f13f9dbd640(0000) GS:ffff9f6be03c0000(0000) knlGS:0000000000000000
Apr 06 12:45:26 tufwrapper kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 06 12:45:26 tufwrapper kernel: CR2: 00000000000000ca CR3: 0000000131212000 CR4: 0000000000f50ef0
Apr 06 12:45:26 tufwrapper kernel: PKRU: 55555554
Apr 06 12:45:26 tufwrapper kernel: note: systemd[993] exited with irqs disabled

Hi @wpkg ,
In your first post you cited an Arch topic and at the end of this topic there is a link to the Nvidia forum and some solutions were given.
Series 550 freezes laptop - #45 by kodatarule - Linux - NVIDIA Developer Forums

Because you did not post the output of the following terminal command:
inxi -v8azy

I could not have a better response, in order to help you to solve the problem.

Regards,

1 Like

Because you did not post the output of the following terminal command:
inxi -v8azy

Thanks for trying to help, I understand that my question is not in the usual forum format where I’m describing a symptom and looking for cause and solutions. In this case I’ve identified the issue as being with Nvidia 550 drivers and have a solution. It’s more of a procedural question of “How do I do … on Manjaro”.

But your point brings to mind an associated idea that forums are not necessarily to solve the problem of an individual but also to act as a reference for future posters who may have the same/similar question.

The hardware I have this issue on is ASUS TUF Gaming F15 (2022), Intel Core i7-12700H 12th Gen, RTX 3050 Ti 4GB Graphics FX577ZE-HN056W. The issue is with Nvidia 550 drivers which causes kernel panic (caps lock flashing) due to some weird interaction with what looks like cgroups during reboot/shutdown and sometimes during start.

I had a hard time figuring out what was going on because panic during reboot/shutdown only shows the last part of the trace i.e. from modules loaded due to limited screen size and being unable to scroll up. Sysrq does not work after kernel panic.

To solve this I downgraded to Nvidia 545 and the kernel panics have disappeared. There is a post on the Nvidia forums and from the people on the post it looks like it’s Arch Linux (and those based on it who are affected - even found a Manjaro user there). Nvidia have a bug open Series 550 freezes laptop - #27 by amrits - Linux - NVIDIA Developer Forums

There is no current update on when this bug would be fixed and I believe 555 release is scheduled for mid May Implement Explicit Sync by amshafer · Pull Request #104 · NVIDIA/egl-wayland · GitHub. It’ll also perhaps bring things like explicit sync which should hopefully make Wayland easier.

This topic was automatically closed 36 hours after the last reply. New replies are no longer allowed.