The problem appears to be that display offloading is not set up. It’s not that the nvidia driver doesn’t work with the 2nd gpu. I’m not aware that display offloading works between the nvidia and the nouveau drivers. I’d suggest you take this issue to the nvidia forum.
Do you have anything relevant in the kernel logs? journalctl -b0 -k | egrep -i '(nvidia|drm|gem)'? Or in the Xorg log? Do the non-functioning monitors detect any input or do they say “no signal”?
RE: My compromise above
I tried modifying the device sections in the /etc/X11/xorg.conf file to be:
Section "Device"
Identifier "Device0"
Driver "nvidia"
VendorName "NVIDIA Corporation"
BoardName "NVIDIA GeForce RTX 2080 Ti"
BusID "PCI:4:0:0"
EndSection
Section "Device"
Identifier "Device1"
Driver "nouveau"
BusID "PCI:3:0:0"
Screen 0
EndSection
Section "Device"
Identifier "Device2"
Driver "nouveau"
BusID "PCI:3:0:0"
Screen 1
EndSection
Section "Device"
Identifier "Device3"
Driver "nouveau"
BusID "PCI:3:0:0"
Screen 2
EndSection
It was just a wild guess at using “nouveau” to get those monitors on the non-nvidia driver.
Didnt work though. 
journalctl -b0 -k | egrep -i '(nvidia|drm|gem)'
Jul 30 19:45:47 vivus kernel: ata6.00: supports DRM functions and may not be fully accessible
Jul 30 19:45:47 vivus kernel: ata6.00: supports DRM functions and may not be fully accessible
Jul 30 19:45:47 vivus kernel: ata9.00: supports DRM functions and may not be fully accessible
Jul 30 19:45:47 vivus kernel: ata9.00: supports DRM functions and may not be fully accessible
Jul 30 19:45:47 vivus kernel: ata10.00: supports DRM functions and may not be fully accessible
Jul 30 19:45:47 vivus kernel: ata10.00: supports DRM functions and may not be fully accessible
Jul 30 19:45:47 vivus systemd[1]: Starting Load Kernel Module drm...
Jul 30 19:45:47 vivus systemd[1]: modprobe@drm.service: Deactivated successfully.
Jul 30 19:45:47 vivus systemd[1]: Finished Load Kernel Module drm.
Jul 30 19:45:47 vivus kernel: audit: type=1400 audit(1627692347.107:3): apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe" pid=352 comm="apparmor_parser"
Jul 30 19:45:47 vivus kernel: audit: type=1400 audit(1627692347.110:4): apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe//kmod" pid=352 comm="apparmor_parser"
Jul 30 19:45:47 vivus kernel: nvidia: module license 'NVIDIA' taints kernel.
Jul 30 19:45:47 vivus kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 510
Jul 30 19:45:47 vivus kernel: nvidia 0000:03:00.0: enabling device (0000 -> 0003)
Jul 30 19:45:47 vivus kernel: nvidia 0000:03:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none
Jul 30 19:45:47 vivus kernel: nvidia 0000:04:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=io+mem
Jul 30 19:45:47 vivus kernel: NVRM: loading NVIDIA UNIX x86_64 Kernel Module 470.57.02 Tue Jul 13 16:14:05 UTC 2021
Jul 30 19:45:47 vivus kernel: input: HDA NVidia HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:10.0/0000:04:00.1/sound/card2/input24
Jul 30 19:45:47 vivus kernel: input: HDA NVidia HDMI/DP,pcm=7 as /devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:10.0/0000:04:00.1/sound/card2/input25
Jul 30 19:45:47 vivus kernel: input: HDA NVidia HDMI/DP,pcm=8 as /devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:10.0/0000:04:00.1/sound/card2/input26
Jul 30 19:45:47 vivus kernel: input: HDA NVidia HDMI/DP,pcm=9 as /devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:10.0/0000:04:00.1/sound/card2/input27
Jul 30 19:45:47 vivus kernel: input: HDA NVidia HDMI/DP,pcm=10 as /devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:10.0/0000:04:00.1/sound/card2/input28
Jul 30 19:45:47 vivus kernel: input: HDA NVidia HDMI/DP,pcm=11 as /devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:10.0/0000:04:00.1/sound/card2/input29
Jul 30 19:45:47 vivus kernel: input: HDA NVidia HDMI/DP,pcm=12 as /devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:10.0/0000:04:00.1/sound/card2/input30
Jul 30 19:45:47 vivus kernel: nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms 470.57.02 Tue Jul 13 16:06:24 UTC 2021
Jul 30 19:45:47 vivus kernel: [drm] [nvidia-drm] [GPU ID 0x00000300] Loading driver
Jul 30 19:45:47 vivus kernel: [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:03:00.0 on minor 0
Jul 30 19:45:47 vivus kernel: [drm] [nvidia-drm] [GPU ID 0x00000400] Loading driver
Jul 30 19:45:47 vivus kernel: [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:04:00.0 on minor 1
Jul 30 19:45:47 vivus kernel: input: HDA NVidia HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:08.0/0000:03:00.1/sound/card1/input17
Jul 30 19:45:47 vivus kernel: input: HDA NVidia HDMI/DP,pcm=7 as /devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:08.0/0000:03:00.1/sound/card1/input18
Jul 30 19:45:47 vivus kernel: input: HDA NVidia HDMI/DP,pcm=8 as /devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:08.0/0000:03:00.1/sound/card1/input19
Jul 30 19:45:47 vivus kernel: input: HDA NVidia HDMI/DP,pcm=9 as /devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:08.0/0000:03:00.1/sound/card1/input20
Jul 30 19:45:47 vivus kernel: input: HDA NVidia HDMI/DP,pcm=10 as /devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:08.0/0000:03:00.1/sound/card1/input21
Jul 30 19:45:47 vivus kernel: input: HDA NVidia HDMI/DP,pcm=11 as /devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:08.0/0000:03:00.1/sound/card1/input22
Jul 30 19:45:47 vivus kernel: input: HDA NVidia HDMI/DP,pcm=12 as /devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:08.0/0000:03:00.1/sound/card1/input23
In the xorg log I see a few things:
[ 177.200] (EE) [drm] Failed to open DRM device for (null): -2
[ 177.200] (EE) [drm] Failed to open DRM device for (null): -2
[ 177.200] (EE) [drm] Failed to open DRM device for (null): -2
[ 177.200] (EE) [drm] Failed to open DRM device for pci:0000:03:00.0: -2
[ 177.200] (EE) [drm] Failed to open DRM device for pci:0000:03:00.0: -2
[ 177.200] (EE) [drm] Failed to open DRM device for pci:0000:03:00.0: -2
I think pci 03 is the card not working. Here’s a couple other interesting lines…
[ 251.760] randr: failed to create shared pixmap
[ 251.760] (EE) NVIDIA(G0): Failed to configure Reverse PRIME on output HDMI-1-0.
Other monitors are no signal. But work fine when I switch the whole system back to the free drivers.
Thanks for the link to the nvidia forums.
That’s just half of the story, you need to care of the kernel drivers as well if you want to go that way.
This is highly likely related, are there other lines like this? Especially ones that mentione “PRIME” and nvidia and randr? What’s the output of sudo cat /sys/module/nvidia_drm/parameters/modeset?