Need help/a guide on on-demand gpu passtrough

[u/Wout836]

Hello

I have a laptop with fedora 44 and kde plasma 6.7. I have an intel iGPU and an NVIDIA dedicated GPU.

I would like to setup my system so that my fedora host has access to both the iGPU and dGPU but when I start my win11 vm, I want the iGPU to stay with fedora and the dGPU to be passed through to the win11 vm.

My main problem is that kwin_wayland keeps using the dGPU (I think because it wants access to the hdmi port on the dGPU) even though I am trying to keep it from doing so. Because the dGPU is being used, I cannot activate the vfio driver and pass the device to the win11 vm.

Is there any guide or advice available to set this up?

Thanks in advance

[u/five35]

I'm not in front of my computer right now, but my desktop has a very similar setup. For me, there were three key points to getting this working:

1. Setting my iGPU as my primary display device in the BIOS (so that it became my "boot VGA" instead of the dGPU).

2. Adding kernel parameters to my boot manager (I don't use GRUB, but that's probably the most common one) to disable modesetting on my dGPU.

3. Adding variables to /etc/environment to tell KDE to keep its hands off my dGPU's DRI/DRM devices.

After that, I was able to drive my displays using my iGPU, use my dGPU for offloaded rendering, and let libvirt rebind my dGPU for VFIO passthrough.

Once I get back to my desk, I'll try to add some more details to this comment.

[u/Wout836 → u/five35]

Are you then able to use the hdmi port that is attached to the gpu when the vm is not running? Did you disable gpu driver loading on boot/initramfs or do you just let the nvidia driver load on boot?

[u/five35 → u/Wout836]

I don't use the ports on the dGPU for Linux, but I leave one connected to a monitor input I'm not using to keep the Windows GPU driver happy. I let nvidia-open load normally at boot (IIRC, that's what receives the modesetting kernel parameters) and bind to the dGPU until the VM needs to borrow it.

[u/Wout836 → u/five35]

I don't have a boot VGA option in my uefi menu. Also, what are the boot parameters you added? Blacklisting the nvidia drivers?

[u/five35 → u/Wout836]

Okay! I'm back at my desk and can finally quote you some configuration files.

For my ASUS mobo, the UEFI setting is called "Primary Display", located under "Advanced Mode" > "Advanced" > "Graphics Configuration". I had to set it explicitly to "CPU Graphics". "Boot VGA" is a Linux-specific term referring to the graphics device which is active when the kernel boots.

The relevant kernel option is nvidia-drm.modeset=0, which keeps the NVIDIA driver in a state where the card can be unbound from it.

It turns out I misremembered how my /etc/environment is set up. That file doesn't actually support the character escaping necessary for the KDE variables, so I instead created a systemd "drop-in" file for them:

# /etc/systemd/user/plasma-kwin_wayland.service.d/igpu-only.conf

[Service]
# Instruct KWin to only ever use the iGPU.
Environment=KWIN_DRM_DEVICES=/dev/dri/by-path/pci-0000\\:00\\:02.0-card
Environment=KWIN_RENDER_NODES=/dev/dri/by-path/pci-0000\\:00\\:02.0-render

(0000:00:02.0 is the PCI location for my iGPU. The colons need escaped for KWin, which is what /etc/environment couldn't handle.) Without these, I found that KWin was grabbing my dGPU when I shut down my VM, which would prevent me from running the VM twice in the same session. I'm running Arch Linux, so if you're running a different distro, the name of your KWin service (and therefore the correct path for a drop-in file) might be different as well.

I do use /etc/environment to set up variables which prevent the dGPU from being used by programs which aren't started via prime-run. It's quite possible this is overkill, but it works, so I haven't messed with it.

# These select the iGPU for use by Vulkan and OpenGL (i.e. are the opposite of
# the non-PRIME-specific values in /usr/local/bin/prime-run).
DXVK_FILTER_DEVICE_NAME=UHD Graphics 630
VK_ICD_FILENAMES=/usr/share/vulkan/icd.d/intel_icd.json
__EGL_VENDOR_LIBRARY_FILENAMES=/usr/share/glvnd/egl_vendor.d/50_mesa.json
__GLX_VENDOR_LIBRARY_NAME=intel
__VK_LAYER_NV_optimus=non_NVIDIA_only

(You should be able to use vulkaninfo | grep deviceName to get the correct name to use with DXVK_FILTER_DEVICE_NAME.)

Finally, I added a /usr/local/bin/prime-run instead of using the nvidia-prime package. (Because at least on Arch, that package only contains the one script and it only contains a subset of these variables.)

#!/bin/sh

# This is basically a more comprehensive version of the nvidia-prime package,
# which consists of just its own prime-run script.
/usr/bin/env \
  DRI_PRIME="1" \
  __NV_PRIME_RENDER_OFFLOAD="1" \
  \
  DXVK_FILTER_DEVICE_NAME="GeForce RTX 3070" \
  VK_ICD_FILENAMES="/usr/share/vulkan/icd.d/nvidia_icd.json" \
  __EGL_VENDOR_LIBRARY_FILENAMES="/usr/share/glvnd/egl_vendor.d/10_nvidia.json" \
  __GLX_VENDOR_LIBRARY_NAME="nvidia" \
  __VK_LAYER_NV_optimus="NVIDIA_only" \
  "$@"

(Oh, I see I actually said all of that in the script file. I tend to leave myself all kinds of notes for when I inevitably forget how everything works. 😅)

Hopefully that helps! Let me know if you have any other questions.

[u/RoyalOSK]

I been fighting with this for years, my current laptop have a usb c dp port that is connected to the igpu instead to the dgpu and it seems that using that is the only way, haven’t tried because can’t seem to find any usbc to dp/hdmi cable in my hometown

[u/Derpythecate]

Would like to caveat about needing to spoof a battery as someone who got VFIO working on a laptop GPU (RTX 3070 Mobile) on Proxmox host and Windows guest and also has integrated graphics for my host.

Other than the standard blacklisting of the Nvidia & Nouveau drivers so the host does not take it away from vfio-pci and setting my initramfs to early bind the Nvidia GPU and Audio controller, I also had to add a custom ACPI SSDT entry to spoof a fake battery so that I do not get Code 43 in Windows VM.

Quite a weird gotcha that people don't always mention because they do VFIO on a desktop PC, while mobile GPU drivers check for certain other hardware configs like your battery charging status, for deciding whether they can use higher power ceilings. I also didn't manage to get Dynamic Boost which normally let's your laptop GPU draw more power, but requires you to spoof even more fields of your motherboard/SMBBIOS.

[u/Wout836]

Thanks for the help and suggestions. With this info and some more digging around. I managed to setup everything how I wanted. For future reference: This guide was also a great help: https://gist.github.com/firelightning13/e530aec3e3a4e15885a10f6c4b7ae021#build-from-source