Amd_iommu=on | NEED HELP

Sure. I think it might be the card thats broken or something… Right now im trying to pass it to VM.

GRUB_DEFAULT=saved
GRUB_TIMEOUT=10
GRUB_TIMEOUT_STYLE=hidden
GRUB_DISTRIBUTOR="Manjaro"
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on  apparmor=1 security=apparmor udev.log_priority=3"
GRUB_CMDLINE_LINUX=""

# If you want to enable the save default function, uncomment the following
# line, and set GRUB_DEFAULT to saved.
GRUB_SAVEDEFAULT=true

# Preload both GPT and MBR modules so that they are not missed
GRUB_PRELOAD_MODULES="part_gpt part_msdos"

# Uncomment to enable booting from LUKS encrypted devices
#GRUB_ENABLE_CRYPTODISK=y

# Uncomment to use basic console
GRUB_TERMINAL_INPUT=console

# Uncomment to disable graphical terminal
#GRUB_TERMINAL_OUTPUT=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command 'videoinfo'
GRUB_GFXMODE=auto

# Uncomment to allow the kernel use the same resolution used by grub
GRUB_GFXPAYLOAD_LINUX=keep

# Uncomment if you want GRUB to pass to the Linux kernel the old parameter
# format "root=/dev/xxx" instead of "root=/dev/disk/by-uuid/xxx"
#GRUB_DISABLE_LINUX_UUID=true

# Uncomment to disable generation of recovery mode menu entries
GRUB_DISABLE_RECOVERY=true

# Uncomment and set to the desired menu colors.  Used by normal and wallpaper
# modes only.  Entries specified as foreground/background.
GRUB_COLOR_NORMAL="light-gray/black"
GRUB_COLOR_HIGHLIGHT="green/black"

# Uncomment one of them for the gfx desired, a image background or a gfxtheme
#GRUB_BACKGROUND="/usr/share/grub/background.png"
GRUB_THEME="/usr/share/grub/themes/manjaro/theme.txt"

# Uncomment to get a beep at GRUB start
#GRUB_INIT_TUNE="480 440 1"

I can’t see that you used vfio or a stub driver. You can’t pass the GPU to a VM, if the host system has already loaded a graphics driver.
Have a look at this.
Then you can take the device IDs of the GPU you want to pass. Also make sure you specify all devices within a IOMMU group.
You can set the kernel parameters like explained here.

Do you mean this???

GNU nano 5.2                                                             /etc/mkinitcpio.conf                                                                        
# vim:set ft=sh
# MODULES
# The following modules are loaded before any boot hooks are
# run.  Advanced users may wish to specify all system modules
# in this array.  For instance:
#     MODULES=(piix ide_disk reiserfs)
MODULES="vfio_pci vfio vfio_iommu_type1 vfio_virqfd"

# BINARIES
# This setting includes any additional binaries a given user may
# wish into the CPIO image.  This is run last, so it may be used to
# override the actual binaries included by a given hook
# BINARIES are dependency parsed, so you may safely ignore libraries
BINARIES=()

# FILES
# This setting is similar to BINARIES above, however, files are added
# as-is and are not parsed in any way.  This is useful for config files.
FILES=""

# HOOKS
# This is the most important setting in this file.  The HOOKS control the
# modules and scripts added to the image, and what happens at boot time.
# Order is important, and it is recommended that you do not change the
# order in which HOOKS are added.  Run 'mkinitcpio -H <hook name>' for
# help on a given hook.
# 'base' is _required_ unless you know precisely what you are doing.
# 'udev' is _required_ in order to automatically load modules
# 'filesystems' is _required_ unless you specify your fs modules in MODULES
# Examples:
##   This setup specifies all modules in the MODULES setting above.
##   No raid, lvm2, or encrypted root is needed.
#    HOOKS=(base modconf)
#
##   This setup will autodetect all modules for your system and should

What file is this?
Also please have a look at the links.

/etc/mkinitcpio.conf

edit

that is exactly what i did…

/etc/modprobe.d/vfio.conf                                                                      
options vfio-pci ids=10de:0a65, 10de:0be3

So i thinks its the gpu itself???

Since these conf files are embedded in the initramfs image, any changes require regenerating a new image each time:

Have you regenerated the initramfs image? I have specified the device IDs via the kernel parameters in /etc/default/grub. Maybe you could try that if it doesn’t work with the initramfs.
With inxi -F you can check if the vfio driver gets loaded for your GPU.

1 Like

I will just buy new GPU and when i get and if it will not work i will create new post. Thanks for helping.

I think the right kernel argument is amd_iommu=on

The right argument is iommu=soft

I would additionally recommend one more argument (to overcome an AMD microcode bug):

processor.max_cstate=5

So your grub.cfg default commandline should read:

" iommu=soft processor.max_cstate=5"

After you make the changes, don’t forget to run update-grub

1 Like

Typo: “I think the right kernel argument is amd_iommu=on”
should have been: “I don’t think the right kernel argument is amd_iommu=on”

1 Like

Would you mind posting the results of inxi -F? Because if vfio isn’t loaded as the graphics driver something is still wrong with the settings where a new GPU would not help either. :wink:

1 Like

Sure here

System:
  Host: s1vde Kernel: 5.8.16-2-MANJARO x86_64 bits: 64 
  Desktop: KDE Plasma 5.19.5 Distro: Manjaro Linux 
Machine:
  Type: Desktop Mobo: ASRock model: B450 Pro4 
  serial: <superuser/root required> UEFI [Legacy]: American Megatrends 
  v: P3.90 date: 12/12/2019 
CPU:
  Topology: 6-Core model: AMD Ryzen 5 2600X bits: 64 type: MT MCP 
  L2 cache: 3072 KiB 
  Speed: 2410 MHz min/max: 2200/3600 MHz Core speeds (MHz): 1: 2712 
  2: 1719 3: 2058 4: 2042 5: 1778 6: 2117 7: 1884 8: 1887 9: 2820 
  10: 1991 11: 1835 12: 2189 
Graphics:
  Device-1: NVIDIA GT218 [GeForce 210] driver: nouveau v: kernel 
  Device-2: AMD Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] 
  driver: amdgpu v: kernel 
  Display: x11 server: X.Org 1.20.9 driver: amdgpu,nouveau FAILED: ati 
  unloaded: modesetting resolution: 1360x768~60Hz 
  OpenGL: renderer: Radeon RX 580 Series (POLARIS10 DRM 3.38.0 
  5.8.16-2-MANJARO LLVM 10.0.1) 
  v: 4.6 Mesa 20.1.8 
Audio:
  Device-1: NVIDIA High Definition Audio driver: snd_hda_intel 
  Device-2: AMD Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590] 
  driver: snd_hda_intel 
  Device-3: AMD Family 17h HD Audio driver: snd_hda_intel 
  Sound Server: ALSA v: k5.8.16-2-MANJARO 
Network:
  Device-1: Realtek RTL8192EE PCIe Wireless Network Adapter 
  driver: rtl8192ee 
  IF: wlp7s0 state: up mac: d0:37:45:99:37:62 
  Device-2: Realtek RTL8111/8168/8411 PCI Express Gigabit Ethernet 
  driver: r8169 
  IF: enp9s0 state: down mac: a8:a1:59:33:55:8d 
  IF-ID-1: virbr0 state: down mac: 52:54:00:a5:98:49 
  IF-ID-2: virbr0-nic state: down mac: 52:54:00:a5:98:49 
Drives:
  Local Storage: total: 1.02 TiB used: 32.91 GiB (3.2%) 
  ID-1: /dev/sda vendor: Kingston model: SV300S37A120G size: 111.79 GiB 
  ID-2: /dev/sdb vendor: Western Digital model: WD10JPCX-24UE4T0 
  size: 931.51 GiB 
Partition:
  ID-1: / size: 109.53 GiB used: 32.91 GiB (30.1%) fs: ext4 
  dev: /dev/sda1 
Swap:
  Alert: No Swap data was found. 
Sensors:
  System Temperatures: cpu: 34.8 C mobo: N/A 
  Fan Speeds (RPM): N/A 
  GPU: device: nouveau temp: 30 C fan: 1470 device: amdgpu temp: 22 C 
  fan: 823 
Info:
  Processes: 323 Uptime: N/A Memory: 15.63 GiB used: 1.30 GiB (8.3%) 
  Shell: Bash inxi: 3.1.05

Thats a problem… I mean i ripped it out like a hour ago so…

edit

i will try again tommorow. I am way too tired today.

Thank you!!! That solved my problems.

edit

iommu=soft didnt work. i replaced it with amd_iommu=on and now everything looks fine?

Graphics:
  Device-1: NVIDIA GT218 [GeForce 210] driver: vfio-pci v: 0.2 
  Device-2: AMD Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] 
  driver: amdgpu v: kernel 
  Display: x11 server: X.Org 1.20.9 driver: amdgpu FAILED: ati 
  unloaded: modesetting resolution: 1360x768~60Hz 
  OpenGL: renderer: Radeon RX 580 Series (POLARIS10 DRM 3.38.0 
  5.8.16-2-MANJARO LLVM 10.0.1) 
  v: 4.6 Mesa 20.1.8 

I will try to pass the NVIDIA GPU. By the looks of it everything should work.

Looks good, lets hope it works. In case it does you will probably have to hide the fact that you are using a VM if you haven’t already to avoid the error 43.

It didnt work… Again it might be the gpu itself

Can you show the generated xml in virt-manager?

How far does it go and what doesn’t work?

VM Doesnt start. It says: Graphical console not configured for guest. Also strange thing i noticed: when i plug in gpu that i passed the screen should be blank right? On my screen it says no signal. I will try to pass rx580 and see if it changes anything.

edit

i added PCI devices to virt manager

XML:

<domain type="kvm">
  <name>win10</name>
  <uuid>e86ff1d5-6410-4af7-9faf-be0c558aa731</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://microsoft.com/win/10"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit="KiB">8290304</memory>
  <currentMemory unit="KiB">8290304</currentMemory>
  <vcpu placement="static">12</vcpu>
  <os>
    <type arch="x86_64" machine="pc-q35-5.1">hvm</type>
    <loader readonly="yes" type="pflash">/usr/share/edk2-ovmf/x64/OVMF_CODE.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/win10_VARS.fd</nvram>
    <boot dev="hd"/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state="on"/>
      <vapic state="on"/>
      <spinlocks state="on" retries="8191"/>
    </hyperv>
    <vmport state="off"/>
  </features>
  <cpu mode="host-model" check="partial">
    <topology sockets="1" dies="1" cores="6" threads="2"/>
  </cpu>
  <clock offset="localtime">
    <timer name="rtc" tickpolicy="catchup"/>
    <timer name="pit" tickpolicy="delay"/>
    <timer name="hpet" present="no"/>
    <timer name="hypervclock" present="yes"/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled="no"/>
    <suspend-to-disk enabled="no"/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type="file" device="disk">
      <driver name="qemu" type="qcow2"/>
      <source file="/var/lib/libvirt/images/win10.qcow2"/>
      <target dev="sda" bus="sata"/>
      <address type="drive" controller="0" bus="0" target="0" unit="0"/>
    </disk>
    <disk type="file" device="cdrom">
      <driver name="qemu" type="raw"/>
      <source file="/home/s1vde/Documents/VMIsoFiles/Windows10 64.iso"/>
      <target dev="sdb" bus="sata"/>
      <readonly/>
      <address type="drive" controller="0" bus="0" target="0" unit="1"/>
    </disk>
    <disk type="file" device="cdrom">
      <driver name="qemu" type="raw"/>
      <source file="/home/s1vde/Documents/VMIsoFiles/virtio-win-0.1.185.iso"/>
      <target dev="sdc" bus="sata"/>
      <readonly/>
      <address type="drive" controller="0" bus="0" target="0" unit="2"/>
    </disk>
    <controller type="usb" index="0" model="qemu-xhci" ports="15">
      <address type="pci" domain="0x0000" bus="0x02" slot="0x00" function="0x0"/>
    </controller>
    <controller type="sata" index="0">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x1f" function="0x2"/>
    </controller>
    <controller type="pci" index="0" model="pcie-root"/>
    <controller type="pci" index="1" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="1" port="0x8"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x0" multifunction="on"/>
    </controller>
    <controller type="pci" index="2" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="2" port="0x9"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x1"/>
    </controller>
    <controller type="pci" index="3" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="3" port="0xa"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x2"/>
    </controller>
    <controller type="pci" index="4" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="4" port="0xb"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x3"/>
    </controller>
    <controller type="pci" index="5" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="5" port="0xc"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x4"/>
    </controller>
    <controller type="pci" index="6" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="6" port="0xd"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x5"/>
    </controller>
    <interface type="network">
      <mac address="52:54:00:5d:5c:66"/>
      <source network="default"/>
      <model type="e1000e"/>
      <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
    </interface>
    <input type="tablet" bus="usb">
      <address type="usb" bus="0" port="1"/>
    </input>
    <input type="mouse" bus="ps2"/>
    <input type="keyboard" bus="ps2"/>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x03" slot="0x00" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x01" slot="0x00" function="0x1"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x04" slot="0x00" function="0x0"/>
    </hostdev>
    <redirdev bus="usb" type="spicevmc">
      <address type="usb" bus="0" port="2"/>
    </redirdev>
    <redirdev bus="usb" type="spicevmc">
      <address type="usb" bus="0" port="3"/>
    </redirdev>
    <memballoon model="virtio">
      <address type="pci" domain="0x0000" bus="0x05" slot="0x00" function="0x0"/>
    </memballoon>
  </devices>
</domain>

You could set the bus type to virtio since you are passing the virtio cd anyways.
You have passed all devices where you loaded the vfio driver to the VM?

Yes, all of em. Even if i would set the bus type to virtio it would not start anyways. I tried passing my rx580 and when i did the screen was blank however when i plugged in my nvidia card to the monitor it didnt post(no signal)…

I will order a new GPU and if it doesnt work i will try to find a solution.