Libvirt: xml and pci addresses assignment

[ghost82]

Hi, I’m trying to understand how to assign a passed through device (vfio) to a predefined address (domain:bus:slot:function).
I’m running manjaro latest stable version with updated libvirt (7.1.0-3) and qemu (6.0.0-2), running a mac os vm q35+ovmf.
As far as I know, for vfio in libvirt we have to specify the source address of the device in the host and a target address for the guest.
So, for example:

<hostdev mode='subsystem' type='pci' managed='yes'>
  <driver name='vfio'/>
  <source>
    <address domain='0x0000' bus='0x00' slot='0x1b' function='0x0'/>
  </source>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
</hostdev>

This will passthrough the device in the host at 0000:00:1b.0 (in the host, domain, bus, slot, function) to 0000:00:02.0 (in the guest, domain, bus, slot, function).

In my xml I have the following devices passed through:

<hostdev mode='subsystem' type='pci' managed='yes'>	// GPU VIDEO [GeForce GTX TITAN Black] [10de:100c]
  <driver name='vfio'/>
  <source>
    <address domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
  </source>
  <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0' multifunction='on'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>	// GPU AUDIO [10de:0e1a]
  <driver name='vfio'/>
  <source>
    <address domain='0x0000' bus='0x04' slot='0x00' function='0x1'/>
  </source>
  <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x1'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>	// MB AUDIO C600/X79 High Definition Audio Controller [8086:1d20]
  <driver name='vfio'/>
  <source>
    <address domain='0x0000' bus='0x00' slot='0x1b' function='0x0'/>
  </source>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>	// SATA Marvell 88SE9230 Controller [1b4b:9230]
  <driver name='vfio'/>
  <source>
    <address domain='0x0000' bus='0x0a' slot='0x00' function='0x0'/>
  </source>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x00' function='0x0' multifunction='on'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>	// USB 3.0 Fresco Logic FL1100 Controller [1b73:1100]
  <driver name='vfio'/>
  <source>
    <address domain='0x0000' bus='0x84' slot='0x00' function='0x0'/>
  </source>
  <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>	// FIREWIRE VIA VT6315 Controller [1106:3403]
  <driver name='vfio'/>
  <source>
    <address domain='0x0000' bus='0x0b' slot='0x00' function='0x0'/>
  </source>
  <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
</hostdev>

Ok, everything shows up in the vm, everything works.

Now, one can stop here, but since I like to understand, I did an lspci -nn in the guest, and this is what is showing:

00:00.0 Host bridge [0600]: Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller [8086:29c0] (subsys 1af4:1100)
00:01.0 PCI bridge [0604]: Red Hat, Inc. (null) [1b36:000c]
00:01.1 PCI bridge [0604]: Red Hat, Inc. (null) [1b36:000c]
00:01.2 PCI bridge [0604]: Red Hat, Inc. (null) [1b36:000c]
00:01.3 PCI bridge [0604]: Red Hat, Inc. (null) [1b36:000c]
00:01.4 PCI bridge [0604]: Red Hat, Inc. (null) [1b36:000c]
01:00.0 PCI bridge [0604]: Red Hat, Inc. (null) [1b36:000e]
00:01.5 PCI bridge [0604]: Red Hat, Inc. (null) [1b36:000c]
03:00.0 Communication controller [0780]: Red Hat, Inc (null) [1af4:1043] (rev 01) (subsys 1af4:1100)
00:02.0 Audio device [0403]: Intel Corporation C600/X79 series chipset High Definition Audio Controller [8086:1d20] (rev 06) (subsys 1043:84d8)
00:02.3 PCI bridge [0604]: Red Hat, Inc. (null) [1b36:000c]
04:00.0 FireWire (IEEE 1394) [0c00]: VIA Technologies, Inc. VT6315 Series Firewire Controller [1106:3403] (rev 01) (subsys 1106:3403)
05:00.0 SATA controller [0106]: Marvell Technology Group Ltd. 88SE9230 PCIe SATA 6Gb/s Controller [1b4b:9230] (rev 10) (subsys 1b4b:9230)
00:07.0 USB controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 [8086:2934] (rev 03) (subsys 1af4:1100)
00:07.7 USB controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 [8086:293a] (rev 03) (subsys 1af4:1100)
00:1f.0 ISA bridge [0601]: Intel Corporation 82801IB (ICH9) LPC Interface Controller [8086:2918] (rev 02) (subsys 1af4:1100)
06:00.0 VGA compatible controller [0300]: NVIDIA Corporation (null) [10de:100c] (rev a1) (subsys 10de:0010)
00:1f.2 SATA controller [0106]: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] [8086:2922] (rev 02) (subsys 1af4:1100)
06:00.1 Audio device [0403]: NVIDIA Corporation GK110 HDMI Audio [10de:0e1a] (rev a1) (subsys 10de:1066)
02:01.0 Ethernet controller [0200]: Red Hat, Inc Virtio network device [1af4:1000] (subsys 1af4:0001)
00:1f.3 SMBus [0c05]: Intel Corporation 82801I (ICH9 Family) SMBus Controller [8086:2930] (rev 02) (subsys 1af4:1100)
02:08.0 Ethernet controller [0200]: Red Hat, Inc Virtio network device [1af4:1000] (subsys 1af4:0001)
08:00.0 USB controller [0c03]: Fresco Logic (null) [1b73:1100] (rev 10) (subsys 1b73:1100)

It doesn’t correspond to what I assigned…

GPU (video) should be at 0000:03:00.0, instead it’s at 0000:06:00.0
GPU (audio) should be at 0000:03:00.1, instead it’s at 0000:06:00.1
Fresco USB should be at 0000:04:00.0 instead it’s at 0000:08:00.0
Firewire should be at 0000:05:00.0 instead it’s at 0000:04:00.0
I have also two network bridges which are at the wrong addresses.
Mainboard audio is correct, it’s specified in the xml at 0000:00:02.0 and shows at 0000:00:02.0
Also the other bridges and virtual usb controller are correct.

I know marvell sata it’s wrong in the xml since 0000:00:00.0 is reserved and hardcoded in qemu, maybe libvirt automatically reassigns it at 000:05:00.0

So, I modified the xml to assign the assigned addresses that showed in the guest (for every vfio devices, for example I assigned for marvell SATA domain=‘0x0000’ bus=‘0x05’ slot=‘0x00’ function=‘0x0’ multifunction=‘on’), but with the result that the lspci command returns (again) not correct addresses.

Anyone is able to explain?

[ghost82]

Nevermind, I think I understood how it works.
From libvirt: PCI addresses in domain XML and guest OS

QEMU, and consequently libvirt, uses the bus property of a device’s PCI address only to match it with the PCI controller that has the same index property, and not to set the actual PCI address, which is decided by the guest OS.

I understand that it’s the guest that assigns the addresses: if we think about a real pc, when you add/remove some hardware, pci addresses of the other hardware can change.

I think we have no control on these addresses, but in real we are able to attach devices to the bus we want: a q35 machine has pcie-root, pcie-root-port and pcie-to-pci-bridge, each of these entries have index=xx, which corresponds to the bus entry in pci address of the device we add; if we choose for example bus=0x00 in the address of the device we are attaching that device to pcie-root (index=0), pcie-root-port is for pcie hot-pluggable devices, pcie-to-pci-bridge is for legacy pci devices.

If I understood well, I can imagine devices attached to pcie-root (index=0 → bus=0x00) as built-in (I attached built-in audio, firewire, ethernet, sata controller and built-in usb uhci/ehci), and to pcie-root-port I attached gpu and it’s audio, a fresco usb controller and virtio-serial. No need for the pcie-to-pci-bridge.

My topology now looks like this:

I think that pcie-root-port attaches to pcie-root, because if I use bus 0, slot 1, function 0 for a device attached to pcie-root the vm fails to start with the message that that address is already in use (by pcie-root-port with index 1).
I also chose I multifunction device for pcie-root-port index 1, so other pcie-root-port are attached to the same slot (different function): not sure what is the difference between a multifunction pcie-root-port vs non multifunction.

[ghost82]

The above was totally wrong, I was editing the vm xml in the wrong way!!
Once edited with virsh edit vmname all is right, lspci in the guest reflects what is specified in the xml.
Moreover, attaching the sata controller to bus=0x00 prevents mac os vm from starting.
Here is the final topology:

and the lspci command in the guest: