Desktop environment not loading while setting up VFIO

Support Virtualization
,
1

On fresh installation of Manjaro KDE, I was doing below steps to setup VFIO

sudo pamac install qemu libvirt ovmf virt-manager  ebtables dnsmasq birdge-utils

Then I edited my GRUB command line to this:

quiet udev.log_priority=3 nvidia_drm.modeset=1 rd.driver.blacklist=nouveau modeprob.blacklist=nouveau video=efifb:off rd.driver.pre=vfio-pci kvm.ignore_msrs=1 vfio-pci.ids=10de:2704,10de:22bb amd_iommu=force_enable iommu=pt hugepages=16384

and updated GRUB using

sudo update-grub

Then I added modules in my mkinit configuration

sudo nano /etc/mkinitcpio.conf
MODULES="vfio vfio_iommu_type1 vfio_pci vfio_virqfd irqbypass kvm kvm_amd xhci_hcd"
HOOKS="base udev autodetect modconf block keyboard keymap filesystems"
sudo mkinitcpio -P

After rebooting I checked my bindings with

lspci -nnk | grep -A 3 -E "VGA|3D"

and verified that iGPU driver is amdgpu and Nvidia driver is vfio-pci

Then I setup network

sudo nano /etc/libvirt/qemu.conf
user=wizard
group=kvm

sudo systemctl enable libvirtd.service
sudo systemctl enable virtlogd.socket
sudo systemctl start libvirtd.service
sudo systemctl start virtlogd.socket

sudo virsh net-define /etc/libvirt/qemu/networks/default.xml
sudo virsh net-autostart default

However after reboot I am stack in booting without loading the desktop environment.

This is where it stops:

[ OK ] Reached target Sound Card.
[ OK ] Started Bluetooth service.
[ OK ] Reached target Bluetooth Support.
Starting Hostname Service...
[ OK ] Started User Login Management.
[ OK ] Started Authorization Manager.
Starting Modem Manager...
[ OK ] Started Hostname Service.
[ OK ] Started Modem Manager.
Starting Network Manager Script Dispatcher Service...
[ OK ] Started Network Manager Script Dispatcher Service.
[ OK ] Started Network Manager.
[ OK ] Reached target Network.
Starting Network Manager Wait Online...
[ OK ] Starting CUPS Scheduler...
[ OK ] Started CUPS Scheduler.
[ OK ] Finished Network Manager Wait Online.
[ OK ] Reached target Network is Online.

[ OK ] Mounted
[ OK ] Mounted
[ OK ] Mounted
[ OK ] Mounted
[ OK ] Mounted
[ OK ] Mounted
[ OK ] Mounted
[ OK ] Mounted
[ OK ] Mounted

[ OK ] Reached target Remote File Systems.
Starting Permit User Sessions...
[ OK ] Finished Permit User Sessions.
[ OK ] Started Command Scheduler.
Starting Hold until boot process finishes up...
Starting Terminate Plymouth Boot Screen...

lspci

[wizard@wizzy-am5-manj-kde6 Wizzy]$ lspci -nnk | grep -A 3 -E "VGA|3D"

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation AD103 [GeForce RTX 4080] [10de:2704] (rev a1)
    Subsystem: Gigabyte Technology Co., Ltd Device [1458:40bc]
    Kernel driver in use: vfio-pci
    Kernel modules: nouveau, nvidia_drm, nvidia

--

15:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Raphael [1002:164e] (rev c9)
    Subsystem: Gigabyte Technology Co., Ltd Device [1458:d000]
    Kernel driver in use: amdgpu
    Kernel modules: amdgpu

[wizard@wizzy-am5-manj-kde6 Wizzy]$

inxi -F

done
[wizard@wizzy-am5-manj-kde6 Wizzy]$ inxi -F

System:
  Host: wizzy-am5-manj-kde6 Kernel: 6.10.13-3-MANJARO arch: x86_64 bits: 64
  Console: tty 2 Distro: Manjaro Linux

Machine:
  Type: Desktop Mobo: Gigabyte model: X670E AORUS MASTER serial: <superuser required>
  UEFI: American Megatrends LLC. v: F33d date: 10/16/2024

CPU:
  Info: 16-core model: AMD Ryzen 9 7950X3D bits: 64 type: MT MCP cache: L2: 16 MiB
  Speed (MHz): avg: 545 min/max: 545/5759 cores: 1: 545 2: 545 3: 545 4: 545 5: 545 6: 545
               7: 545 8: 545 9: 545 10: 545 11: 545 12: 545 13: 545 14: 545 15: 545 16: 545
               17: 545 18: 545 19: 545 20: 545 21: 545 22: 545 23: 545 24: 545 25: 545 26: 545
               27: 545 28: 545 29: 545 30: 545 31: 545 32: 545

Mod edit: Images removed and replaced with text. Last one will be OCR’d and added once I have time.

2

are you sure about:
modeprob.blacklist=nouveau
?

Without looking it up I would guess it should be:
modprobe.blacklist=nouveau

3

Yes, you are right, but it is correct in command line. Just a typo when writing this topic.

4

Perhaps this is Plymouth related.

You could try disabling (or removing) Plymouth to see if that makes a difference. See the REMOVAL heading in Plymouth Manjaro Wiki article for specific instructions.

This might (or might not) resolve the booting issue.
I’m unable to help with VFIO.

Good luck.

5

Thank you for your suggestion.
For some reason, after a fresh install, I was stuck in the splash screen, so I always remove “splash” from GRUB. I re-added it to make sure that wasn’t the problem, since I am removing it on first boot.
Didn’t help.
Then I removed it, along with “quiet” and added “text” and also uninstalled plymouth.

It didn’t help. I am stuck here

Image
Image1920Ă—569 122 KB

Some entries from logs:

Image-2
Image-21920Ă—780 263 KB

Image-1
Image-11920Ă—408 146 KB

6

Your journalctl (and/or) dmesg should have more useful information than your boot up screenshots.

Are you sure the vfio module is getting loaded first?

Nvidia users may also have to use:
/etc/modprobe.d/vfio.conf :

softdep nvidia pre: vfio-pci

What is your libvirt config, where you expose your IOMMU groups? Are they 100% accurate?

I have this working (on a desktop anyway), but there is too much information missing to really tell what is going on.

7

The lack on information is because I didn’t have access to internet, while in tty2 and copy/paste flexibility. I have a workaround (had used it before), and I can boot to desktop, but cannot run VMs.

Tell me what you need and I can upload it.

softdep nvidia pre: vfio-pci

I can try this, but not sure how it will help. If you see my 1st post, vfio-pci grabs the Nvidia, but then it does not load the environment. So, I think the problem is elsewhere.

My IOMMU groups

IOMMU Group 0:
        00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14da]
IOMMU Group 1:
        00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14db]
IOMMU Group 10:
        00:08.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14dd]
IOMMU Group 11:
        00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 71)
        00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU Group 12:
        00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e0]
        00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e1]
        00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e2]
        00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e3]
        00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e4]
        00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e5]
        00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e6]
        00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e7]
IOMMU Group 13:
        01:00.0 VGA compatible controller [0300]: NVIDIA Corporation AD103 [GeForce RTX 4080] [10de:2704] (rev a1)
        01:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:22bb] (rev a1)
IOMMU Group 14:
        02:00.0 Non-Volatile memory controller [0108]: Kingston Technology Company, Inc. KC3000/FURY Renegade NVMe SSD E18 [2646:5013] (rev 01)
IOMMU Group 15:
        03:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Upstream Port [1022:43f4] (rev 01)
IOMMU Group 16:
        04:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
IOMMU Group 17:
        04:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
IOMMU Group 18:
        04:05.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
IOMMU Group 19:
        04:06.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
IOMMU Group 2:
        00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14db]
IOMMU Group 20:
        04:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
        09:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Upstream Port [1022:43f4] (rev 01)
        0a:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
        0a:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
        0a:06.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
        0a:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
        0a:0c.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
        0a:0d.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
        0c:00.0 Ethernet controller [0200]: Intel Corporation 82576 Gigabit Network Connection [8086:10c9] (rev 01)
        0c:00.1 Ethernet controller [0200]: Intel Corporation 82576 Gigabit Network Connection [8086:10c9] (rev 01)
        0e:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I225-V [8086:15f3] (rev 01)
        0f:00.0 Non-Volatile memory controller [0108]: ADATA Technology Co., Ltd. XPG GAMMIX S50, S50 Lite NVMe SSD [1cc1:5350] (rev 03)
        10:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset USB 3.2 Controller [1022:43f7] (rev 01)
        11:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset SATA Controller [1022:43f6] (rev 01)
IOMMU Group 21:
        04:0c.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
        12:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset USB 3.2 Controller [1022:43f7] (rev 01)
IOMMU Group 22:
        04:0d.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
        13:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset SATA Controller [1022:43f6] (rev 01)
IOMMU Group 23:
        14:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 [144d:a808]
IOMMU Group 24:
        15:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Raphael [1002:164e] (rev c9)
IOMMU Group 25:
        15:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Family 19h PSP/CCP [1022:1649]
IOMMU Group 26:
        15:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15b6]
IOMMU Group 27:
        15:00.4 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15b7]
IOMMU Group 28:
        15:00.6 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Family 17h/19h HD Audio Controller [1022:15e3]
IOMMU Group 29:
        16:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15b8]
IOMMU Group 3:
        00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14da]
IOMMU Group 4:
        00:02.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14db]
IOMMU Group 5:
        00:02.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14db]
IOMMU Group 6:
        00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14da]
IOMMU Group 7:
        00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14da]
IOMMU Group 8:
        00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14da]
IOMMU Group 9:
        00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14dd]

As for libvirt.conf, I haven’t made any changes to it, so it is just a dummy file with commented info

8

I meant the guest libvirt config. e.g.:

sudo virsh dumpxml $DOMAIN > domain.xml

Then you can copy it to a flash drive, scp it to another host (that’s not Internet), or whatever.

But this is your host having trouble booting right? So the problem shouldn’t lie in there. But just given what it’s doing, the VFIO/GPU module binding seems very likely still.

That small change I got right from: https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF#modprobe.d

(Which I assume you followed?)

This is where the (nicely formatted) logs might help…

Just two to start:

dmesg | grep -i vfio
journalctl --boot 0 --priority 3 --no-pager

First is the kernel output just grepping for vfio, and the second is all errors and up in your logs, from when you powered on or restarted last.

Once you have the VFIO stuff all configured, then you will need to setup your iGPU to work if it wasn’t already. This is another possibility where the problem lies. And where the actual logs may give a hint.

At this point, you would be like a Manjaro install that had a GPU ripped out, that possibly might still see it, but not able to access it. I’ve heard different things for this. I would even try installing your AMD iGPU drivers (again) via sudo mhwd -i at this point, if you haven’t. I would guess it would be using video-hybrid-amd-nvidia-prime now? (But you don’t want Nvidia or hybrid/Prime.)

9

A workaround (for reference):

In a TTY you can still pipe command output to a text file; and then copy that text file to a more convenient location (perhaps a USB).

  • Using > will create a new file or overwrite the existing file.
  • Using >> will append more output to the same file.

For example:

journalctl --boot=-1 --priority=3 --catalog --no-pager > myoutput.txt
10

First of all, thank you both for your time. I have this issue for months, and I need to solve it.
Previously I had an AlienWare monitor, and the workaround was easy, I just had to turn on my monitor after the system was on the login screen, then everything worked smoothly. Minimum trouble.
However, I had to return that monitor 3 times, so I asked for a refund and got another one. That trick no longer works and I have to remove the cable from Nvidia, which is under the desk, and only plug it back, after I start a VM that has Nvidia passed through.
I believe there is some sort of active link in the DP port, that causes the system to think the monitor is present, even though it is switched of, something that AlienWare had implement differently.

@Molski you are right, nothing to do with my domain XML, I haven’t even tried to create one yet.

So, this is my dmesg & journal from when I boot without the cable attached.

[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-6.10-x86_64 root=UUID=2e8c7577-24b7-4ee5-be84-fe6a3b9da786 rw quiet udev.log_priority=3 vfio-pci.ids=10de:2704,10de:22bb video=efifb:off kvm.ignore_msrs=1 amd_iommu=force_enable iommu=pt hugepages=16384
[    0.059800] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-6.10-x86_64 root=UUID=2e8c7577-24b7-4ee5-be84-fe6a3b9da786 rw quiet udev.log_priority=3 vfio-pci.ids=10de:2704,10de:22bb video=efifb:off kvm.ignore_msrs=1 amd_iommu=force_enable iommu=pt hugepages=16384
[    4.599237] VFIO - User Level meta-driver version: 0.3
[    4.607132] vfio-pci 0000:01:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=io+mem:owns=none
[    4.607216] vfio_pci: add [10de:2704[ffffffff:ffffffff]] class 0x000000/00000000
[    4.654219] vfio_pci: add [10de:22bb[ffffffff:ffffffff]] class 0x000000/00000000
[    8.888004] Modules linked in: amdgpu(+) nouveau hid_generic drm_gpuvm mxm_wmi amdxcp i2c_algo_bit drm_ttm_helper ttm usbhid drm_suballoc_helper drm_exec drm_buddy gpu_sched nvme crc32c_intel drm_display_helper nvme_core xhci_pci xhci_pci_renesas cec video nvme_auth wmi vfio_pci vfio_pci_core vfio_iommu_type1 vfio iommufd
[   24.332273] vfio-pci 0000:01:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=io+mem:owns=none

Nov 06 14:35:40 wizzy-am5-manj-kde6 kernel: i2c i2c-1: Failed to register i2c client MSFT8000:00 at 0x40 (-16)
Nov 06 14:35:40 wizzy-am5-manj-kde6 kernel: hub 10-0:1.0: config failed, hub doesn't have any ports! (err -19)
Nov 06 14:35:40 wizzy-am5-manj-kde6 kernel: usb 3-5.4: 11:3: cannot get min/max values for control 2 (id 11)
Nov 06 14:35:40 wizzy-am5-manj-kde6 kernel: usb 3-5.4: 13:3: cannot get min/max values for control 2 (id 13)
Nov 06 14:35:40 wizzy-am5-manj-kde6 kernel: usb 3-5.4: 12:3: cannot get min/max values for control 2 (id 12)
Nov 06 14:35:40 wizzy-am5-manj-kde6 kernel: usb 3-5.4: 16:3: cannot get min/max values for control 2 (id 16)
Nov 06 14:35:40 wizzy-am5-manj-kde6 kernel: usb 3-5.4: 1:3: cannot get min/max values for control 2 (id 1)
Nov 06 14:35:40 wizzy-am5-manj-kde6 kernel: usb 3-5.4: 2:3: cannot get min/max values for control 2 (id 2)
Nov 06 14:35:40 wizzy-am5-manj-kde6 kernel: usb 3-5.4: 23:3: cannot get min/max values for control 2 (id 23)
Nov 06 14:35:40 wizzy-am5-manj-kde6 kernel: usb 3-5.4: 25:3: cannot get min/max values for control 2 (id 25)
Nov 06 14:35:40 wizzy-am5-manj-kde6 kernel: usb 3-5.4: 8:3: cannot get min/max values for control 2 (id 8)
Nov 06 14:35:40 wizzy-am5-manj-kde6 kernel: usb 3-5.4: 27:3: cannot get min/max values for control 2 (id 27)
Nov 06 14:35:41 wizzy-am5-manj-kde6 (udev-worker)[728]: controlC1: /usr/lib/udev/rules.d/78-sound-card.rules:5 Failed to write ATTR{/sys/devices/pci0000:00/0000:00:02.1/0000:03:00.0/0000:04:0c.0/0000:12:00.0/usb3/3-5/3-5.4/3-5.4:1.3/sound/card1/controlC1/../uevent}="change", ignoring: No such file or directory
Nov 06 14:35:41 wizzy-am5-manj-kde6 bluetoothd[904]: Failed to set mode: Failed (0x03)
Nov 06 14:42:11 wizzy-am5-manj-kde6 org_kde_powerdevil[1585]: busno=2, sleep-multiplier =  1.30. Testing for supported feature 0x10 returned Error_Info[EIO in ddc_write_read_with_retry, causes: EIO]
Nov 06 14:42:11 wizzy-am5-manj-kde6 org_kde_powerdevil[1585]: busno=7, sleep-multiplier =  2.00. Testing for supported feature 0x10 returned Error_Info[DDCRC_RETRIES in ddc_write_read_with_retry, causes: EREMOTEIO(10)]
Nov 06 14:42:11 wizzy-am5-manj-kde6 bluetoothd[904]: Failed to set mode: Failed (0x03)
Nov 06 14:42:12 wizzy-am5-manj-kde6 bluetoothd[904]: Failed to set mode: Failed (0x03)

And these two below, are from when I am stuck before DE in tty2

[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-6.10-x86_64 root=UUID=2e8c7577-24b7-4ee5-be84-fe6a3b9da786 rw quiet udev.log_priority=3 vfio-pci.ids=10de:2704,10de:22bb video=efifb:off kvm.ignore_msrs=1 amd_iommu=force_enable iommu=pt hugepages=16384
[    0.060074] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-6.10-x86_64 root=UUID=2e8c7577-24b7-4ee5-be84-fe6a3b9da786 rw quiet udev.log_priority=3 vfio-pci.ids=10de:2704,10de:22bb video=efifb:off kvm.ignore_msrs=1 amd_iommu=force_enable iommu=pt hugepages=16384
[    4.270738] VFIO - User Level meta-driver version: 0.3
[    4.337397] vfio-pci 0000:01:00.0: vgaarb: deactivate vga console
[    4.337404] vfio-pci 0000:01:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=io+mem:owns=none
[    4.337576] vfio_pci: add [10de:2704[ffffffff:ffffffff]] class 0x000000/00000000
[    4.383925] vfio_pci: add [10de:22bb[ffffffff:ffffffff]] class 0x000000/00000000
[    8.581338] Modules linked in: amdgpu(+) hid_generic nouveau drm_gpuvm amdxcp mxm_wmi i2c_algo_bit drm_ttm_helper ttm usbhid drm_suballoc_helper drm_exec gpu_sched drm_buddy nvme crc32c_intel drm_display_helper nvme_core xhci_pci xhci_pci_renesas cec nvme_auth video wmi vfio_pci vfio_pci_core vfio_iommu_type1 vfio iommufd

Nov 06 14:55:18 wizzy-am5-manj-kde6 kernel: i2c i2c-1: Failed to register i2c client MSFT8000:00 at 0x40 (-16)
Nov 06 14:55:18 wizzy-am5-manj-kde6 kernel: hub 10-0:1.0: config failed, hub doesn't have any ports! (err -19)
Nov 06 14:55:19 wizzy-am5-manj-kde6 bluetoothd[893]: Failed to set mode: Failed (0x03)
Nov 06 14:56:10 wizzy-am5-manj-kde6 sddm[1077]: Failed to read display number from pipe
Nov 06 14:56:14 wizzy-am5-manj-kde6 sddm[1077]: Failed to read display number from pipe
Nov 06 14:56:17 wizzy-am5-manj-kde6 sddm[1077]: Failed to read display number from pipe
Nov 06 14:56:17 wizzy-am5-manj-kde6 sddm[1077]: Could not start Display server on vt 2
11

I was hoping for more… But those commands do not encompass everything. Look at your entire dmesg, and every journalctl entry around those key times.

To me, there for be a chance your AMD iGPU could just be the problem. (Just move forward as if the dGPU doesn’t exist.) Why I suggested just trying the quick Manjaro install GPU driver helper script.

You said yourself it was broken even before, until you attached a display until after the login screen loaded?

12

To make things clear:
It works fine with proprietary drivers.
Works OK with open source, mouse is laggy and hangs often.
This phenomenon starts as soon as I add “vfio_pci vfio vfio_iommu_type1” in MODULES section of /etc/mkinitcpio.conf.
I have the same behaviour with both OS and proprietary drivers.
Removing the cable works with both OS and proprietary drivers. What I am doing is detaching the cable from Nvidia, work and when I want to use a VM with the Nvidia passed through, I boot the VM, wait to see the desktop and then I attach the cable. VM then get the signal and uses the Vnidia card normally.

I have tried with other distros (openSuse, EndeavourOS, Kubuntu) and I have the same issue, therefore I believe it us Kernel/Driver issue.

The only distro I tried that works with just 3 steps, is Fedora, but I prefer Manjaro all the way.

13

Some working (and non-working) configuration examples can be found here; perhaps some of these might reveal something useful:

Good luck.

14

Thank you but I have seen that. None is Ryzen 9 with iGPU as host…
I had it working before, with other setups, even with this one, on Kernel 5.x.
The problem started with Kernel 6.x.

The only thing I don’t fully understand, is this step from the Arch wiki:

If you are modesetting the nvidia driver, the vfio-pci.ids must be embedded in the initramfs image. If given via kernel arguments, they will be read too late to take effect. Follow the instructions in #Binding vfio-pci via device ID for adding the ids to a modprobe conf file.

15

If you are modesetting the nvidia driver, the vfio-pci.ids must be embedded in the initramfs image. If given via kernel arguments, they will be read too late to take effect. Follow the instructions in #Binding vfio-pci via device ID for adding the ids to a modprobe conf file.

Why are you modesetting it? This gives it access to APIs and DRM type stuff. But we want to modeset it in the guest. (I have tried to hand off my Nvidia GPU from host to guest, but that was at least a couple years ago.) But even then, I could not do this without very experimental code and custom GPU firmware. So I just leave it uninitialized, and bound to the vfio module until a guest picks it up.

The Arch wiki for this was much different 3 years ago when I set it up. I use kernel parameters just fine (though I would do it the other way, if it worked).

Part of the reason when earlier, I suggested that pre load dependency config first, as it’s what the Arch wiki said. But did you try via kernel parameter? I mean it works for me. (On a desktop anyway.)

I even have a AMD host GPU, and Nvidia guest:

./iommu_groups.sh | grep -E 'NVID|ATI'
IOMMU Group 24 04:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Cedar [Radeon HD 5000/6000/7350/8350 Series] [1002:68f9]
IOMMU Group 24 04:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cedar HDMI Audio [Radeon HD 5400/6300/7300 Series] [1002:aa68]
IOMMU Group 29 0c:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA102 [GeForce RTX 3080 Lite Hash Rate] [10de:2216] (rev a1)
IOMMU Group 29 0c:00.1 Audio device [0403]: NVIDIA Corporation GA102 High Definition Audio Controller [10de:1aef] (rev a1)

grep -i CMDLINE /etc/default/grub | head -1
GRUB_CMDLINE_LINUX_DEFAULT="quiet nosplash apparmor=1 security=apparmor resume=UUID=9f5a3a86-21ad-4bcc-9db7-2377aee3d05c udev.log_priority=3 amd_iommu=on vfio-pci.ids=10de:2216,10de:1aef video=efifb:off isolcpus=6-11,18-23 fbcon=rotate:1"

That works with Windows, other *NIXs, and every other guest I’ve tried.

(I also pass a USB controller so I can plug in USB devices directly into certain VMs without SPICE, but you don’t need to hide it from the kernel here.)

I only got a laptop capable of doing this a couple months ago. It was on my list of things to try, I did expect problems before even attempting it. Laptops aren’t like desktops, as they share display outputs. And they aren’t true separate PCIe devices as they are in desktops. At the same time, so many more tools to help this, that exist now where they didn’t when I set it up.

e.g.: envycontol and supergfxctl (which I see even supports vfio?)