Nvidia render offloading

As reported by Phoronix, Nvidia has added render offloading to the beta 435.17 driver.
Official documentation: http://download.nvidia.com/XFree86/Linux-x86_64/435.17/README/primerenderoffload.html

What does this mean?
No more fudging around with optirun, xrun or switching GPUs to play games on Optimus notebooks.

[WIP] How do I get it?
You need to install some packages from AUR:
yay -Syu xorg-server-git nvidia-full-beta-all nvidia-utils-full-beta-all lib32-nvidia-utils-full-beta-all opencl-nvidia-full-beta-all

The opencl package isn't really necessary, unless you use some program that requires it.

I was using optimus manager, so remove it first:
yay -R optimus-manager

Allow nvidia-drm to load on boot by commenting stuff on
sudo nano /etc/modprobe.d/mhwd-nvidia.conf

Now add some lines to xorg.conf
sudo nano /etc/X11/xorg.conf.d/01-nvidia-offloading.conf

Section "ServerFlags"
  Option "IgnoreABI" "1"
EndSection

Section "ServerLayout"
  Identifier "layout"
  Option "AllowNVIDIAGPUScreens"
  Screen 0 "iGPU"
EndSection

Section "Device"
  Identifier "iGPU"
  Driver "modesetting"
  BusID "PCI:0:2:0"
EndSection

Section "Screen"
  Identifier "iGPU"
  Device "iGPU"
EndSection

Section "Device"
  Identifier "nvidia"
  Driver "nvidia"
  BusID "PCI:1:0:0"
EndSection

In the documentation it says it should work out of the box, but I had to add the ignoreabi flag or the driver would refuse to load, after that the desktop would render to the dGPU, so I added the rest with the PCI ids to force to render the desktop on the iGPU.

If everything goes well it should be working already after a reboot. You can run vulkan programs on the dGPU with:
__NV_PRIME_RENDER_OFFLOAD=1 vkcube
And openGL programs with:
__NV_PRIME_RENDER_OFFLOAD=1 __GLX_VENDOR_LIBRARY_NAME=nvidia glxinfo

For wine, even if it uses dxvk, it's best to use both.

The only problem I found is that it doesn't power down the GPU, from what I read this only works on Turing (I have a 1050 Pascal).

6 Likes

We will add this officially to our unstable branch later today.

6 Likes

everything works, but i was only able to get the xorg/desktop running only on modesetting once, im gonna try again from a clean slate. i cant make power management work if a xorg process is running on the nvidia.

1 Like

@petete

are you able to boot into an igpu only setting on the modesetting driver (no xorg process shown in nvidia-smi), and then from there be able to run something graphical using the nvidia?

i tried both with aur packages as you posted earlier, then removed those and installed the manjaro packages with the same outcome on each.

using the xorg config you posted, if i only blacklist nouveau, xorg will run the process on the nvidia card but rendering done on the intel which is pointless since the nvidia card cant be powered down, nor is it doing anything other than running and 11mb xorg process. but i am able to start vulkan and gl processes on the nvidia gpu.

the only way im able to get xorg running only on the modesetting driver is to blacklist the nvidia modules from being loaded during boot, which is both good and bad, the nvidia gpu powers itself down to 0%, then turns on once i modprobe nvidia nvidia-drm but im unable to run anything on it.

nvidia modules blacklisted
[dglt@dglt-i3 ~]$ lsmod|grep nvidia
[dglt@dglt-i3 ~]$ inxi -Gxxxz
Graphics:  Device-1: Intel HD Graphics 530 vendor: Dell driver: i915 v: kernel bus ID: 00:02.0 chip ID: 8086:191b 
           Device-2: NVIDIA GM107M [GeForce GTX 960M] vendor: Dell driver: N/A bus ID: 02:00.0 chip ID: 10de:139b 
           Display: x11 server: X.Org 1.20.5 driver: modesetting unloaded: nvidia compositor: compton 
           resolution: 1920x1080~60Hz 
           OpenGL: renderer: Mesa DRI Intel HD Graphics 530 (Skylake GT2) v: 4.5 Mesa 19.1.4 compat-v: 3.0 direct render: Yes 
[dglt@dglt-i3 ~]$ sudo modprobe nvidia nvidia-drm
[sudo] password for dglt: 
[dglt@dglt-i3 ~]$ inxi -Gxxxz
Graphics:  Device-1: Intel HD Graphics 530 vendor: Dell driver: i915 v: kernel bus ID: 00:02.0 chip ID: 8086:191b 
           Device-2: NVIDIA GM107M [GeForce GTX 960M] vendor: Dell driver: nvidia v: 435.17 bus ID: 02:00.0 chip ID: 10de:139b 
           Display: x11 server: X.Org 1.20.5 driver: modesetting unloaded: nvidia compositor: compton 
           resolution: 1920x1080~60Hz 
           OpenGL: renderer: Mesa DRI Intel HD Graphics 530 (Skylake GT2) v: 4.5 Mesa 19.1.4 compat-v: 3.0 direct render: Yes 
[dglt@dglt-i3 ~]$ 
[dglt@dglt-i3 ~]$ nvidia-smi
Wed Aug 14 20:59:20 2019       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 435.17       Driver Version: 435.17       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 960M    Off  | 00000000:02:00.0 Off |                  N/A |
| N/A   47C    P0    N/A /  N/A |      0MiB /  4046MiB |      1%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
[dglt@dglt-i3 ~]$ __NV_PRIME_RENDER_OFFLOAD=1 __GLX_VENDOR_LIBRARY_NAME=nvidia glxinfo
name of display: :0
X Error of failed request:  BadValue (integer parameter out of range for operation)
  Major opcode of failed request:  152 (GLX)
  Minor opcode of failed request:  24 (X_GLXCreateNewContext)
  Value in failed request:  0x0
  Serial number of failed request:  39
  Current serial number in output stream:  40
[dglt@dglt-i3 ~]$ 
[dglt@dglt-i3 ~]$ __NV_PRIME_RENDER_OFFLOAD=1 vkcube
Segmentation fault (core dumped)
[dglt@dglt-i3 ~]$ 

not blacklisted
[dglt@dglt-i3 ~]$ lsmod|grep nvidia
nvidia_drm             57344  2
nvidia_modeset       1126400  2 nvidia_drm
nvidia              19554304  78 nvidia_modeset
drm_kms_helper        225280  2 nvidia_drm,i915
drm                   507904  8 drm_kms_helper,nvidia_drm,i915
ipmi_msghandler        69632  2 ipmi_devintf,nvidia
[dglt@dglt-i3 ~]$ inxi -Gxxxz
Graphics:  Device-1: Intel HD Graphics 530 vendor: Dell driver: i915 v: kernel bus ID: 00:02.0 chip ID: 8086:191b 
           Device-2: NVIDIA GM107M [GeForce GTX 960M] vendor: Dell driver: nvidia v: 435.17 bus ID: 02:00.0 chip ID: 10de:139b 
           Display: x11 server: X.Org 1.20.5 driver: modesetting,nvidia compositor: compton resolution: 1920x1080~60Hz 
           OpenGL: renderer: Mesa DRI Intel HD Graphics 530 (Skylake GT2) v: 4.5 Mesa 19.1.4 compat-v: 3.0 direct render: Yes 
[dglt@dglt-i3 ~]$ nvidia-smi
Wed Aug 14 21:30:38 2019       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 435.17       Driver Version: 435.17       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 960M    Off  | 00000000:02:00.0 Off |                  N/A |
| N/A   33C    P8    N/A /  N/A |     22MiB /  4046MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0       840      G   /usr/lib/Xorg                                 21MiB |
+-----------------------------------------------------------------------------+
[dglt@dglt-i3 ~]$ echo "with glxgears and vkcube running"
with glxgears and vkcube running
[dglt@dglt-i3 ~]$ echo "with glxgears and vkcube running"
with glxgears and vkcube running
[dglt@dglt-i3 ~]$ 
[dglt@dglt-i3 ~]$ nvidia-smi
Wed Aug 14 21:39:09 2019       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 435.17       Driver Version: 435.17       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 960M    Off  | 00000000:02:00.0 Off |                  N/A |
| N/A   33C    P8    N/A /  N/A |     61MiB /  4046MiB |     30%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0       840      G   /usr/lib/Xorg                                 49MiB |
|    0      1440      G   glxgears                                       2MiB |
|    0      1448    C+G   vkcube                                         6MiB |
+-----------------------------------------------------------------------------+
[dglt@dglt-i3 ~]$ 

can you confirm that behavior?

Xorg will use both GPUs even if using intel as main, so you should still see it on nvidia-smi. Look at the BusID settings. Without it X would try to make the nvidia GPU the default one for me. You can find the correct IDs for your hardware with lspci.
00:02.0 VGA compatible controller: Intel Corporation HD Graphics 630 (rev 04) 01:00.0 3D controller: NVIDIA Corporation GP107M [GeForce GTX 1050 Mobile] (rev a1)

Also be sure to check xrandr --listproviders too, you should see something like this:

Providers: number : 2 Provider 0: id: 0x1df cap: 0xf, Source Output, Sink Output, Source Offload, Sink Offload crtcs: 3 outputs: 4 associated providers: 0 name:modesetting Provider 1: id: 0x1b8 cap: 0x0 crtcs: 0 outputs: 0 associated providers: 0 name:NVIDIA-G0

using the xorg config you posted, if i only blacklist nouveau, xorg will run the process on the nvidia card but rendering done on the intel which is pointless since the nvidia card cant be powered down, nor is it doing anything other than running and 11mb xorg process. but i am able to start vulkan and gl processes on the nvidia gpu.

Well that's the point of render offloading. The intel GPU is used by default, but you can run demanding stuff on the nvidia side. The card powers down only on turing from what i read, but it might change in the future. Anyway it might still save some battery as the nvidia gpu is on the lowest power setting all the time. Also using the intel drivers feel like it is less buggy on my desktop (KDE plasma).

right, and im able to make everything function fine, both providers are listed and the intel gpu/mesa is listed as the opengl/renderer. and using the prime offload variables will run them on the nvidia gpu.

now this is why im confused because the way nvidia and your instructions alike have it setup, have a xorg process running on the nvidia card so it's not possible for the nvidia gpu to power down. this doesnt seem right to me because even turing cards would not be able to power down if they have processes running on them. :thinking:

i can make this work by killing xorg >> unload modules >> start xorg only on the intel gpu. but unless im missing something should not be needed.

I stopped and removed Bumblebee, removed video-hybrid-intel-nvidia-bumblebee mhwd config, reinstalled video-linux config (for some reason this was related to nvidia driver failure to load), installed linux52-nvidia and linux53-nvidia, blacklisted nouveau (since it causes severe issues on reboot), added OP's Xorg config but without bus IDs and IgnoreABI option.
Everything works fine on my MX150, it's a pity that it doesn't power down though.
Nvidia-settings looks weird now:

[$] <> glxinfo |grep vendor    
server glx vendor string: SGI
client glx vendor string: Mesa Project and SGI
OpenGL vendor string: Intel Open Source Technology Center
[$] <> __NV_PRIME_RENDER_OFFLOAD=1 __GLX_VENDOR_LIBRARY_NAME=nvidia glxinfo |grep vendor
server glx vendor string: NVIDIA Corporation
client glx vendor string: NVIDIA Corporation
OpenGL vendor string: NVIDIA Corporation
[$] <> mhwd -li
> Installed PCI configs:
--------------------------------------------------------------------------------
                  NAME               VERSION          FREEDRIVER           TYPE
--------------------------------------------------------------------------------
           video-linux            2018.05.04                true            PCI
1 Like

you shouldnt need video-linux installed at all.

As I understand, yes, I shouldn't need it, but glamoregl failed to initialize until video-linux was installed, preventing nvidia from loading. The only part of video-linux which was missing at that point was xf86-video-nouveau, maybe that was exactly what I needed? However, nouveau was immediately blacklisted with kernel parameter as it causes hard lockups.
I think I need to explore this more.

1 Like

im hoping to figure something out, the way it is now makes little sense. if the nvidia needs to remain powered on to run a xorg process it defeats the purpose of switchable graphics. yes, the nvidia gpu would be utilized less than it would with a full prime setup but just being powered on alone even in it's lowest state still consumes a non negligible amount of power.

that, and using the modesetting driver to render is absolutely abysmal at best. tearing doesnt even begin to describe it. i cant imagine this is the intended functionality though so thats why im wondering if this is setup incorrectly. if the nvidia card needs to remain powered on to run a xorg process then you might as well just use the nvidia for rendering as well. it's only a few days old though so it's too early to tell what the intended outcome is. documentation on it is still minimal.

As I see it now, it may be possible to use such way of setting graphics switching:

  1. kill xorg
  2. run bbswitch to unload nvidia module(s) and set something like nouveau.modeset=1 to switch over to nouveau or load it if blacklisted.

I mean use mixed Bumblebee / optimus-switch approach.
It's a shame I'm not a developer.

PS: I've just discovered that kernel nouveau driver does not cause issues like xf86-video-nouveau.

im able to do this, minus the bbswitch and nouveau parts. killing xorg, unloading modules and starting xorg on only the intel gpu allows this to happen, as long as all the nvidia modules are unloaded and the nvidia gpu power management set to "Auto" the nvidia gpu will power off. for some, an acpi call might be needed.

i was hopeful that this new feature would allow for switching without killing xorg or rebooting at all but i guess thats not possible regardless of what model gpu you have. theres no way around starting xorg on the nvidia gpu for this feature to work, so to be able to power it off you have to kill xorg.

the best possible outcome using this feature without restarting xorg is getting the nvidia gpu to use as little power as possible and unless you have a turing gpu theres not much you can do about it besides adjust clock/mem frequencies and keeping it in its lowest performance state, even the turing cards can not be powered off and instead they have better ability to go into lower power states and so are more efficient when not under load.

i guess this would be ok for some, it's a step in the right direction for nvidia on linux but where it's a deal breaker for me is where it makes the least amount of sense to use it in it's current state.

  • nvidia always on, but all the rendering is done by the modesetting driver and it's spectacularly bad at that job.
  • because of this, PRIME sync does not work and the entire session has a nasty tearing problem and that also includes processes started using the nvidia launch variables.
  • so at the moment this particular feature you get tearing regardless of which gpu you start the process.
  • you still pay the premium price of "PRIME" but get performance of an etch-a-sketch.

so for now, it's only on my test install i created just to play with this. i'm excited to see where it goes from here and as is my stubborn nature i will still try and make it usable.

on the plus side, it seems at least for me anyway that installing the nvidia packages (not fumblebee) alone without even using a /etc/X11/xorg.conf.d/*.conf results in a perfectly bootable system whereas before would certainly end in a black screen so thats a huge improvement IMO and allows for dropping fumblebee altogether when the user select "nonfree" driver.

Does this affect NVIDIA PRIME at all or should prime users disregard?
[HowTo] Set up PRIME with NVIDIA proprietary driver

Section "Device"
    Identifier "intel"
    Driver "modesetting"
    BusID  "PCI:0:2:0"
    Option "AccelMethod" "none"
EndSection

If so, is glamor now required in the intel config section for modesetting?

prime still works fine, im using it now

1 Like

Yes, working fine here as well. Are you using AccelMethod glamor for Intel? or none? And do we require any xorg conf changes?

this is my current setup when using prime
https://github.com/dglt1/optimus-switch/blob/master/switch/nvidia/nvidia-xorg.conf

with prime, you dont set the accell method because the nvidia gpu is doing all the work and using the intel gpu only to display it. whereas with render offload, compositing is done via the intel using the modesetting driver.

both options use both nvidia and modesetting drivers, just in different ways. heres some comparison outputs from both prime and render offload on my laptop

PRIME setup: https://pastebin.com/e4SU9A62

Render Offload: https://pastebin.com/ECs23UjM

but to answer your question, no changes are needed to continue using prime.

1 Like

Very good, I appreciate the comparison outputs also.

Hmm, with this config, my external monitor (connected over DisplayPort over USB-C) doesn't work, which isn't surprising if I'm on the Intel card. Wondering if I can tweak something to make it work with this config. I don't know much about X.org config and am not sure what to learn.

...oh. Seems that the env vars don't even work, so perhaps I don't have the correct packages (sorry for the fish):

 ~  glxinfo | grep vendor                                                                                                                          14:39:28
server glx vendor string: SGI
client glx vendor string: Mesa Project and SGI
OpenGL vendor string: Intel Open Source Technology Center
 ~  env __NV_PRIME_RENDER_OFFLOAD=1 glxinfo | grep vendor                                                                                          14:39:39
server glx vendor string: SGI
client glx vendor string: Mesa Project and SGI
OpenGL vendor string: Intel Open Source Technology Center
 ~  env __NV_PRIME_RENDER_OFFLOAD=1 glxinfo | grep vendor                                                                                          14:40:16
server glx vendor string: SGI
client glx vendor string: Mesa Project and SGI
OpenGL vendor string: Intel Open Source Technology Center
 ~  env __NV_PRIME_RENDER_OFFLOAD=1 __GLX_VENDOR_LIBRARY_NAME=nvidia glxinfo | grep vendor                                                         14:40:19
X Error of failed request:  BadValue (integer parameter out of range for operation)
  Major opcode of failed request:  152 (GLX)
  Minor opcode of failed request:  24 (X_GLXCreateNewContext)
  Value in failed request:  0x0
  Serial number of failed request:  39
  Current serial number in output stream:  40

I'm on the testing branch which had seemed to get nvidia-435.17-2 and xorg-server-1.20.5-2, but perhaps I have to wait a bit longer. I will switch back to optimus-manager for now and see what they wind up implementing (or if a new mhwd-nvidia config comes along that does this for me). I'm willing to test something else if it would help, though. At least I can still get into KDE, even if on the wrong screen...