Sharing experiences with working Nvidia Optimus, HiDPI (4k), Nvidia-Exec, Fractional Scaling on Gnome

nika · 19 August 2022 00:38

Hi all,

I just wanted to share my working setup with Nvidia Optimus, HiDPI (4k), Nvidia-Exec on Gnome. I am pretty happy with it - although I wish it didnt involve any manual steps (see below).

I hope it can be useful and maybe inspire anyone else to use a similar setup - or help me improve or develop tooling!

Setup

HP Zbook Studio G5 with UHD display (HDMI is routed through a Nvidia Quadro Pascal GPU)
External monitor 27 inch with 1440p resolution and 144hz
Running Gnome 42.4, Nvidia 515.65.01-6, mutter-x11-scaling 42.4-2 (AUR (en) - mutter-x11-scaling), nvidia-exec 0.1.0-4 (AUR (en) - nvidia-exec)

Usage Scenarios
I can switch between those with logging out of Gnome (no reboot needed, but some manual steps, see below)

on the go - hybrid optimus setup
I use Gnome Wayland for on the go. If I need to run an application on the GPU I use Nvidia Exec. For example, to edit pictures running opencl on the GPU I can use ‘nvx start darktable’. Games work to, for example I can start X-Plane with the --vulkan switch.

Advantages:

Low power consumption, even with scaling (I use 175 scaling and get 5-7h battery life)
Nvidia available if needed, with full Vulkan support and no bumblebee penalty (as far as I can judge)

Disadvantages:

HDMI port is unavailable (monitor turns on but is not activated in Gnome, see logs below)
Sometimes chromium grabs my GPU while it is active, but I mostly use firefox

Nvidia stationary setup
I start Gnome on X11 (config below). My external monitor is recognized and I can use both internal and external monitor.

Advantages

Can use external monitor, runs at 144hz
Mixing UHD laptop screen and QHD external monitor works (thanks mutter-x11-scaling!)

Disadvantages

Fans are running, I notice it gets better when I close my laptop and only external screen is running (I believe it might be connected to a large virtual buffer that mutter-x11-scaling creates)
I have to run X11 (would prefer to always be on Wayland)

Transition:

If I want to run scenario 2, i run ‘nvx on’ and logout + log into ‘Gnome on Xorg’
If I want to go back to scenario 1, i log off and while in GDM, I open a virtual terminal and run:

sudo systemctl stop gdm
nvx off
sudo systemctl start gdm

I then chose ‘Gnome’ to start my wayland session again

Of course I would prefer to not have to do those manual steps, but I haven’t found the time to do a script. I would not be sure how to handle the logouts gracefully. But it might actually be quite simple…

Configuration

/etc/X11/xorg.conf.d/20-intel.conf:

Section "Device"
        Identifier  "Intel Graphics"
        Driver      "intel"
        #Option      "DRI" "2"             # DRI3 is now default 
        #Option      "AccelMethod"  "sna" # default
        #Option      "AccelMethod"  "uxa" # fallback
EndSection

Section "ServerLayout"
    Identifier  "Layout0"
    Option      "AutoAddDevices" "true" 
    Option      "AutoAddGPU" "false"
EndSection

Section "Device"
    Identifier  "DiscreteNvidia"
    Driver      "nvidia"
    VendorName  "NVIDIA Corporation"
    Option "ProbeAllGpus" "false"
    Option "NoLogo" "true"
    Option "UseEDID" "true"
    Option "AllowEmptyInitialConfiguration"
#   Option "UseDisplayDevice" "none"
EndSection

Section "Screen"
    Identifier "Screen0"
    Device "DiscreteNvidia"
EndSection

I dont remember whether it was necessary, but I set WaylandEnable=true in /etc/gdm/custom.conf and also ran sudo ln -s /dev/null /etc/udev/rules.d/61-gdm.rules

I dont use nvidia_drm.modeset=1 since it causes gnome-shell to grab the external GPU and doesn’t allow to release it

External monitor in Wayland

I notice that when I enable modesetting, I can use the external screen in Wayland, but I loose the ability to turn off the nvidia (it is blocked by gnome-shell and thus nvx off hangs)

Without modesetting, here is what the logs show if I try to connect the external monitor after running nvx on while in a Wayland session (I think I tried twice below):

Aug 18 23:00:40 nk-zbook sudo[7051]:     nils : TTY=pts/0 ; PWD=/home/nils ; USER=root ; COMMAND=/usr/bin/tee /sys/bus/pci/devices/0000:00:01.0/power/control
Aug 18 23:00:40 nk-zbook sudo[7051]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1000)
Aug 18 23:00:40 nk-zbook sudo[7051]: pam_unix(sudo:session): session closed for user root
Aug 18 23:00:40 nk-zbook dbus-daemon[882]: [system] Activating via systemd: service name='org.freedesktop.home1' unit='dbus-org.freedesktop.home1.service' requested by ':1.277' (uid=0 pid=7068 comm="sudo tee /s>
Aug 18 23:00:40 nk-zbook dbus-daemon[882]: [system] Activation via systemd failed for unit 'dbus-org.freedesktop.home1.service': Unit dbus-org.freedesktop.home1.service not found.
Aug 18 23:00:40 nk-zbook sudo[7068]: pam_systemd_home(sudo:account): systemd-homed is not available: Unit dbus-org.freedesktop.home1.service not found.
Aug 18 23:00:40 nk-zbook sudo[7068]:     nils : TTY=pts/0 ; PWD=/home/nils ; USER=root ; COMMAND=/usr/bin/tee /sys/bus/pci/devices/0000:01:00.0/power/control
Aug 18 23:00:40 nk-zbook sudo[7068]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1000)
Aug 18 23:00:40 nk-zbook sudo[7068]: pam_unix(sudo:session): session closed for user root
Aug 18 23:00:40 nk-zbook dbus-daemon[882]: [system] Activating via systemd: service name='org.freedesktop.home1' unit='dbus-org.freedesktop.home1.service' requested by ':1.278' (uid=0 pid=7076 comm="sudo modpro>
Aug 18 23:00:40 nk-zbook dbus-daemon[882]: [system] Activation via systemd failed for unit 'dbus-org.freedesktop.home1.service': Unit dbus-org.freedesktop.home1.service not found.
Aug 18 23:00:40 nk-zbook sudo[7076]: pam_systemd_home(sudo:account): systemd-homed is not available: Unit dbus-org.freedesktop.home1.service not found.
Aug 18 23:00:40 nk-zbook sudo[7076]:     nils : TTY=pts/0 ; PWD=/home/nils ; USER=root ; COMMAND=/usr/bin/modprobe nvidia
Aug 18 23:00:40 nk-zbook sudo[7076]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1000)
Aug 18 23:00:42 nk-zbook kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 508
Aug 18 23:00:42 nk-zbook kernel: 
Aug 18 23:00:42 nk-zbook kernel: nvidia 0000:01:00.0: enabling device (0000 -> 0003)
Aug 18 23:00:42 nk-zbook kernel: nvidia 0000:01:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none
Aug 18 23:00:42 nk-zbook kernel: NVRM: loading NVIDIA UNIX x86_64 Kernel Module  515.65.01  Wed Jul 20 14:00:58 UTC 2022
Aug 18 23:00:42 nk-zbook systemd-udevd[7037]: nvidia: Process '/usr/bin/bash -c '/usr/bin/mknod -Z -m 666 /dev/nvidiactl c $(grep nvidia-frontend /proc/devices | cut -d \  -f 1) 255'' failed with exit code 1.
Aug 18 23:00:42 nk-zbook systemd-udevd[7083]: nvidia: Process '/usr/bin/bash -c '/usr/bin/mknod -Z -m 666 /dev/nvidiactl c $(grep nvidia-frontend /proc/devices | cut -d \  -f 1) 255'' failed with exit code 1.
Aug 18 23:00:42 nk-zbook sudo[7076]: pam_unix(sudo:session): session closed for user root
Aug 18 23:00:42 nk-zbook systemd-udevd[7083]: nvidia: Process '/usr/bin/bash -c 'for i in $(cat /proc/driver/nvidia/gpus/*/information | grep Minor | cut -d \  -f 4); do /usr/bin/mknod -Z -m 666 /dev/nvidia${i}>
Aug 18 23:00:42 nk-zbook dbus-daemon[882]: [system] Activating via systemd: service name='org.freedesktop.home1' unit='dbus-org.freedesktop.home1.service' requested by ':1.279' (uid=0 pid=7109 comm="sudo modpro>
Aug 18 23:00:42 nk-zbook dbus-daemon[882]: [system] Activation via systemd failed for unit 'dbus-org.freedesktop.home1.service': Unit dbus-org.freedesktop.home1.service not found.
Aug 18 23:00:42 nk-zbook sudo[7109]: pam_systemd_home(sudo:account): systemd-homed is not available: Unit dbus-org.freedesktop.home1.service not found.
Aug 18 23:00:42 nk-zbook sudo[7109]:     nils : TTY=pts/0 ; PWD=/home/nils ; USER=root ; COMMAND=/usr/bin/modprobe nvidia_uvm
Aug 18 23:00:42 nk-zbook sudo[7109]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1000)
Aug 18 23:00:43 nk-zbook systemd-udevd[7037]: nvidia: Process '/usr/bin/bash -c 'for i in $(cat /proc/driver/nvidia/gpus/*/information | grep Minor | cut -d \  -f 4); do /usr/bin/mknod -Z -m 666 /dev/nvidia${i}>
Aug 18 23:00:43 nk-zbook kernel: nvidia_uvm: module uses symbols nvUvmInterfaceDisableAccessCntr from proprietary module nvidia, inheriting taint.
Aug 18 23:00:44 nk-zbook kernel: nvidia-uvm: Loaded the UVM driver, major device number 506.
Aug 18 23:00:44 nk-zbook sudo[7109]: pam_unix(sudo:session): session closed for user root
Aug 18 23:00:44 nk-zbook systemd-udevd[7036]: nvidia_uvm: Process '/usr/bin/bash -c '/usr/bin/mknod -Z -m 666 /dev/nvidia-uvm c $(grep nvidia-uvm /proc/devices | cut -d \  -f 1) 0'' failed with exit code 1.
Aug 18 23:00:44 nk-zbook systemd-udevd[7036]: nvidia_uvm: Process '/usr/bin/bash -c '/usr/bin/mknod -Z -m 666 /dev/nvidia-uvm-tools c $(grep nvidia-uvm /proc/devices | cut -d \  -f 1) 1'' failed with exit code >
Aug 18 23:00:44 nk-zbook dbus-daemon[882]: [system] Activating via systemd: service name='org.freedesktop.home1' unit='dbus-org.freedesktop.home1.service' requested by ':1.280' (uid=0 pid=7132 comm="sudo modpro>
Aug 18 23:00:44 nk-zbook dbus-daemon[882]: [system] Activation via systemd failed for unit 'dbus-org.freedesktop.home1.service': Unit dbus-org.freedesktop.home1.service not found.
Aug 18 23:00:44 nk-zbook sudo[7132]: pam_systemd_home(sudo:account): systemd-homed is not available: Unit dbus-org.freedesktop.home1.service not found.
Aug 18 23:00:44 nk-zbook sudo[7132]:     nils : TTY=pts/0 ; PWD=/home/nils ; USER=root ; COMMAND=/usr/bin/modprobe nvidia_modeset
Aug 18 23:00:44 nk-zbook sudo[7132]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1000)
Aug 18 23:00:44 nk-zbook kernel: nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms  515.65.01  Wed Jul 20 13:43:59 UTC 2022
Aug 18 23:00:44 nk-zbook sudo[7132]: pam_unix(sudo:session): session closed for user root
Aug 18 23:00:44 nk-zbook systemd-udevd[7036]: nvidia_modeset: Process '/usr/bin/bash -c '/usr/bin/mknod -Z -m 666 /dev/nvidia-modeset c $(grep nvidia-frontend /proc/devices | cut -d \  -f 1) 254'' failed with e>
Aug 18 23:00:44 nk-zbook dbus-daemon[882]: [system] Activating via systemd: service name='org.freedesktop.home1' unit='dbus-org.freedesktop.home1.service' requested by ':1.281' (uid=0 pid=7146 comm="sudo modpro>
Aug 18 23:00:44 nk-zbook dbus-daemon[882]: [system] Activation via systemd failed for unit 'dbus-org.freedesktop.home1.service': Unit dbus-org.freedesktop.home1.service not found.
Aug 18 23:00:44 nk-zbook sudo[7146]: pam_systemd_home(sudo:account): systemd-homed is not available: Unit dbus-org.freedesktop.home1.service not found.
Aug 18 23:00:44 nk-zbook sudo[7146]:     nils : TTY=pts/0 ; PWD=/home/nils ; USER=root ; COMMAND=/usr/bin/modprobe nvidia_drm
Aug 18 23:00:44 nk-zbook sudo[7146]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1000)
Aug 18 23:00:44 nk-zbook sudo[7146]: pam_unix(sudo:session): session closed for user root
Aug 18 23:00:44 nk-zbook kernel: [drm] [nvidia-drm] [GPU ID 0x00000100] Loading driver
Aug 18 23:00:44 nk-zbook kernel: [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:01:00.0 on minor 1
Aug 18 23:00:44 nk-zbook gnome-shell[5352]: Failed to open atomic modesetting backend: GDBus.Error:System.Error.ENODEV: No such device
Aug 18 23:00:44 nk-zbook gnome-shell[5352]: g_hash_table_destroy: assertion 'hash_table != NULL' failed
Aug 18 23:00:44 nk-zbook gnome-shell[5352]: Failed to open legacy modesetting backend: GDBus.Error:System.Error.ENODEV: No such device
Aug 18 23:00:44 nk-zbook gnome-shell[5352]: Failed to hotplug secondary gpu '/dev/dri/renderD129': No suitable mode setting backend found
Aug 18 23:00:44 nk-zbook gnome-shell[5352]: g_hash_table_destroy: assertion 'hash_table != NULL' failed
Aug 18 23:00:44 nk-zbook gnome-shell[5352]: Failed to hotplug secondary gpu '/dev/dri/card1': No suitable mode setting backend found
Aug 18 23:01:01 nk-zbook CROND[7210]: (root) CMD (run-parts /etc/cron.hourly)
Aug 18 23:01:01 nk-zbook dbus-daemon[882]: [system] Activating via systemd: service name='org.opensuse.Snapper' unit='snapperd.service' requested by ':1.282' (uid=0 pid=7215 comm="snapper --config=root --quiet >
Aug 18 23:01:01 nk-zbook systemd[1]: Starting DBus interface for snapper...
Aug 18 23:01:01 nk-zbook dbus-daemon[882]: [system] Successfully activated service 'org.opensuse.Snapper'
Aug 18 23:01:01 nk-zbook systemd[1]: Started DBus interface for snapper.
Aug 18 23:01:02 nk-zbook CROND[7209]: (root) CMDEND (run-parts /etc/cron.hourly)
Aug 18 23:01:04 nk-zbook gnome-shell[5352]: Window manager warning: last_user_time (413002) is greater than comparison timestamp (412954).  This most likely represents a buggy client sending inaccurate timestam>
Aug 18 23:01:04 nk-zbook gnome-shell[5352]: Window manager warning: W2 appears to be one of the offending windows with a timestamp of 413002.  Working around...

Open to any ideas or improvements. Let me know if you have any questions of course!

Yochanan · 19 August 2022 01:21

Your post will be helpful, thanks! I just received my first hybrid Intel+NVIDIA laptop in fact, it’s a bit different.

I’ve never run across nvidia-exec before. I use external monitors all the time, so I probably won’t have to worry about switching much. However, I do have switcheroo-control installed for that in case I do. It’s an optional dependency of gnome-control-center now.

nika · 19 August 2022 09:20

Hi @Yochanan!

I think most recent laptops (turing and onwards) are probably easier, and it also makes a difference if your HDMI / displayport outputs are wired through your Nvidia, like it is the case for me.

Do you run wayland or X11?

Cheers

Yochanan · 19 August 2022 14:48

They are. The Thunderbolt port is managed by Intel. I use three monitors and usually keep the laptop lid closed. All monitors support 144Hz, however I had to set the Thunderbolt monitor to 60Hz, otherwise it would blink on and off. I have PRIME Synchronization enabled, it may have something to do with that.

X11 for now. I haven’t tried Wayland on this laptop yet. I tried it from time to time on my previous laptop that had dedicated NVIDIA graphics, but hardware acceleration in browsers and games didn’t work well. Wayland works great on my backup laptop with an AMD APU.

nika · 9 October 2022 08:43

Actually it also works on wayland, but requires modeset to be set both on NVIDIA and intel modules.

Yochanan · 9 October 2022 18:14

FYI, Wayland is not enabled by default if hybrid graphics are detected in GNOME 43:

/usr/lib/udev/rules.d/61-gdm.rules

# If this is a hybrid graphics laptop with vendor nvidia driver, disable wayland
LABEL="gdm_hybrid_nvidia_laptop_check"
TEST!="/run/udev/gdm-machine-is-laptop", GOTO="gdm_hybrid_nvidia_laptop_check_end"
TEST!="/run/udev/gdm-machine-has-hybrid-graphics", GOTO="gdm_hybrid_nvidia_laptop_check_end"
TEST!="/run/udev/gdm-machine-has-vendor-nvidia-driver", GOTO="gdm_hybrid_nvidia_laptop_check_end"
GOTO="gdm_disable_wayland"
LABEL="gdm_hybrid_nvidia_laptop_check_end"

nika · 11 October 2022 13:53

Got it, but Fedora actually enabled it in Fedora36 AFAIK? Changes/WaylandByDefaultOnNVIDIA - Fedora Project Wiki

It works actually quite nice - except with pop shell gnome extension I seem to be getting some random crashes - and is much quieter and battery lasts longer as it doesn’t run the whole UI on nvidia like when I use X11 - which might be a config issue. That is despite not fully turning off due to pre-turing nvidia.

Yochanan · 11 October 2022 14:56

They did. They use a patch now with their gdm 43.0 package for Fedora 37.