Error when trying to pass my RTX 3080 to a VM

Hi, i am trying to pass my RTX 3080, unsurprisingly its not working i am getting this error when creating the VM.

Unable to complete install: 'internal error: qemu unexpectedly closed the monitor: 2020-10-02T00:01:15.125028Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
2020-10-02T00:01:15.127699Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12]
2020-10-02T00:01:15.127739Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
2020-10-02T00:01:15.130352Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12]
2020-10-02T00:01:15.130391Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
2020-10-02T00:01:15.202958Z qemu-system-x86_64: -device vfio-pci,host=0000:01:00.0,id=hostdev0,bus=pci.4,addr=0x0: vfio 0000:01:00.0: group 1 is not viable
Please ensure all devices within the iommu_group are bound to their vfio bus driver.'

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 75, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/createvm.py", line 2089, in _do_async_install
    guest.installer_instance.start_install(guest, meter=meter)
  File "/usr/share/virt-manager/virtinst/install/installer.py", line 542, in start_install
    domain = self._create_guest(
  File "/usr/share/virt-manager/virtinst/install/installer.py", line 491, in _create_guest
    domain = self.conn.createXML(install_xml or final_xml, 0)
  File "/usr/lib/python3.8/site-packages/libvirt.py", line 4035, in createXML
    if ret is None:raise libvirtError('virDomainCreateXML() failed', conn=self)
libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2020-10-02T00:01:15.125028Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
2020-10-02T00:01:15.127699Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12]
2020-10-02T00:01:15.127739Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
2020-10-02T00:01:15.130352Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12]
2020-10-02T00:01:15.130391Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
2020-10-02T00:01:15.202958Z qemu-system-x86_64: -device vfio-pci,host=0000:01:00.0,id=hostdev0,bus=pci.4,addr=0x0: vfio 0000:01:00.0: group 1 is not viable
Please ensure all devices within the iommu_group are bound to their vfio bus driver.

Some more info

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:2206] (rev a1) <- this is my rtx 3080 vga
	Subsystem: eVga.com. Corp. Device [3842:3885]
	Kernel driver in use: vfio-pci
	Kernel modules: nouveau, nvidia_drm, nvidia
01:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:1aef] (rev a1) <- this is my rtx 3080 audio
	Subsystem: eVga.com. Corp. Device [3842:3885]
	Kernel driver in use: vfio-pci
	Kernel modules: snd_hda_intel

Here is the list of devices in my IOMMU group 1

00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x16) [8086:1901] (rev 07)
00:01.1 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x8) [8086:1905] (rev 07)

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:2206] (rev a1) <- vfio
01:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:1aef] (rev a1) <- vfio

02:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU106 [GeForce RTX 2070] [10de:1f02] (rev a1)
02:00.1 Audio device [0403]: NVIDIA Corporation TU106 High Definition Audio Controller [10de:10f9] (rev a1)
02:00.2 USB controller [0c03]: NVIDIA Corporation TU106 USB 3.1 Host Controller [10de:1ada] (rev a1)
02:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU106 USB Type-C UCSI Controller [10de:1adb] (rev a1)

Here is information about the PC
5.4.67-1-MANJARO

Motherboard: Z390 AORUS ULTRA (VT-d is set to enabled in the bios, i am also on the latest firmware version which is F10b)

CPU: Intel core i7 8700
GPUS: (Both plugged straight in the motherboard)

  • rtx 3080 (pcie x16 top slot)

  • rtx 2070 (pcie x8 2nd slot)

Memory: 16gb dual channel

i also tried to pass my RTX 2070 with all its USB controllers and i got the same error.

Is that done?

What do you mean by all devices ? why should i need to pass my rtx 2070 that i use for my main display? if you look in my post the rtx 3080 only has 2 devices and both of them are bound to vfio, you can also see that in my post.

Please run

for g in /sys/kernel/iommu_groups/*; do
    echo "IOMMU Group ${g##*/}:"
    for d in $g/devices/*; do
        echo -e "\t$(lspci -nns ${d##*/})"
    done;
done;

if I recall correctly, all devices shown in the same IOMMU group as the graphics card must be passed through to the VM.

IOMMU Group 0:
	00:00.0 Host bridge [0600]: Intel Corporation 8th Gen Core Processor Host Bridge/DRAM Registers [8086:3ec2] (rev 07)
IOMMU Group 1:
	00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x16) [8086:1901] (rev 07)
	00:01.1 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x8) [8086:1905] (rev 07)
	01:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:2206] (rev a1)
	01:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:1aef] (rev a1)
	02:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU106 [GeForce RTX 2070] [10de:1f02] (rev a1)
	02:00.1 Audio device [0403]: NVIDIA Corporation TU106 High Definition Audio Controller [10de:10f9] (rev a1)
	02:00.2 USB controller [0c03]: NVIDIA Corporation TU106 USB 3.1 Host Controller [10de:1ada] (rev a1)
	02:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU106 USB Type-C UCSI Controller [10de:1adb] (rev a1)
IOMMU Group 10:
	00:1f.0 ISA bridge [0601]: Intel Corporation Z390 Chipset LPC/eSPI Controller [8086:a305] (rev 10)
	00:1f.3 Audio device [0403]: Intel Corporation Cannon Lake PCH cAVS [8086:a348] (rev 10)
	00:1f.4 SMBus [0c05]: Intel Corporation Cannon Lake PCH SMBus Controller [8086:a323] (rev 10)
	00:1f.5 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH SPI Controller [8086:a324] (rev 10)
	00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (7) I219-V [8086:15bc] (rev 10)
IOMMU Group 2:
	00:12.0 Signal processing controller [1180]: Intel Corporation Cannon Lake PCH Thermal Controller [8086:a379] (rev 10)
IOMMU Group 3:
	00:14.0 USB controller [0c03]: Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller [8086:a36d] (rev 10)
	00:14.2 RAM memory [0500]: Intel Corporation Cannon Lake PCH Shared SRAM [8086:a36f] (rev 10)
IOMMU Group 4:
	00:14.3 Network controller [0280]: Intel Corporation Wireless-AC 9560 [Jefferson Peak] [8086:a370] (rev 10)
IOMMU Group 5:
	00:16.0 Communication controller [0780]: Intel Corporation Cannon Lake PCH HECI Controller [8086:a360] (rev 10)
IOMMU Group 6:
	00:17.0 RAID bus controller [0104]: Intel Corporation SATA Controller [RAID mode] [8086:2822] (rev 10)
IOMMU Group 7:
	00:1b.0 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #17 [8086:a340] (rev f0)
IOMMU Group 8:
	00:1c.0 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #1 [8086:a338] (rev f0)
IOMMU Group 9:
	00:1d.0 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #9 [8086:a330] (rev f0)

If i pass my 2070 i wont get any output on my manjaro

I think you need to somehow isolate the RTX 3080 (plugging it into a different port) so that the devices you want to pass through are in their own IOMMU group.

That is not possible not enough clearance. there must be another way?

Do you have IOMMU setting in Bios and if so what is it set to?

I have an intel cpu its called vt-d and its set to enabled.

OK if you are sure you enabled the proper setting. I’m pretty sure your Bios settings are missing something, make sure you review all Bios settings only you can see how it is configured.

I managed to separate the gpu in there own iommu groups using something called acs patch but now when i start the vm the screen truns off and stays black, i have a keyboard mouse connected to it and i also added

<vendor_id state="on" value="0123456789ab"/>
and
<kvm>
  <hidden state="on"/>
</kvm>

in the xml since im passing an nvidia gpu but it still only shows a black screen. anyone have a idea?

Go to 9:45 and he shows you step by step on how to get your graphics card to pass through

Hopefully, it will help

Your guide does not get me further than where i already am.

Did you try google? because I find many results, like this one

i fixed all issues here is how i did

I compiled a custom kernel with the acs patch and set pcie_acs_override=downstream in grub_cmdline_linux_default, after that the gpu were in seperate groups but i got a black screen i fixed that by installing the os before passing the gpu, then passing the gpu and installing the drivers (from the spice deisplay) after doing that the screen will still be black to fix this you have to add this to your grub_cmdline_linux “video=astdrmfb video=efifb:off”

after doing that everyting worked fine here is my config.

cpu: i7-8700 (not k)
mobo: z390 aorus ultra (set vt-d to enabled if you have intel cpu)
gpus:

  • rtx 3080 (guest, connected to first pcie slot)
  • rtx 2070 (host, connected to 2nd pcie slot) <- make sure to set the bios to start up on this pcie slot
    ram: 16gb 2666k (im upgrading soon don’t worry)

os:

  • manjaro 20.1 downgraded to linux54 (host)
  • windows 10 home (guest)

if you are using this setup you need to make sure you have kvm hidden in the xml file of you kvm.

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.