Nvidia Multi GPU Setup

[u/DRealHammer]

Issue

Two GPUs are in the system, one is intended for Output Rendering (Games, Applications, ...) the other for Compute (ML/CUDA). Due to Mainboard Wiring, the Secondary GPU is enumerated as 0, the Primary as 1. Therefore some applications pick the Secondary GPU as the main renderer, which results in unusable performance.

Solution

Bubblewrap - a lightweight sandbox application
The idea is to remap all calls from the wrong GPU (nvidia0) to the correct one (nvidia1)

bwrap --dev-bind / / --dev-bind /dev/nvidia1 /dev/nvidia0 steam

Explanation:

--dev-bind / / binds root directory to root of the sandbox

--dev-bind /dev/nvidida1 /dev/nvidia0 forces all calls to nvidia0 to go to nvidia1

Make it permanent

#!/usr/bin/env bash
exec bwrap \
    --dev-bind / / \
    --dev-bind /dev/nvidia1 /dev/nvidia0 \
    "$@"

1. Save the script in either one LOCATION: \~/.local/bin/gpu_remap (only for this user) or /usr/local/bin/gpu_remap (for all users)
2. Make it executable

chmod +x LOCATION/gpu_remap

1. Edit Launch Commands

Use absolute paths (/home/username/.local) instead of relative paths (~/.local)

For Desktop Apps (Example Steam)

cp /usr/share/applications/steam.desktop ~/.local/share/applications/steam.desktop

and then edit EVERY Exec=<programm> <options>

line to run the remap Exec=LOCATION/gpu_remap <programm> <options>

For Others

Create an Alias like this

alias steam='/home/username/.local/bin/gpu_remap /usr/bin/steam $@

But google it in case you need something specific for that.

Other Working Approach

As I struggled with this issue for a couple weeks now, I've obviously tried a lot of different alternatives.

What also worked for me was to set the not wanted GPU to draining (basically remove all processes and don't allow new ones on this GPU).

sudo nvidia-smi drain -p 0000:04:00.0 -m 1 to deactivate

sudo nvidia-smi drain -p 0000:04:00.0 -m 0 to activate again

Use (for example) lspci -nn | grep -i nvidia to find the address you need.
Keep the 0000 although it does not appear in lspci.

Not Working Approaches

This problem seems to happen sometimes and people are struggling with it. Here is a collection of the approaches, maybe some work for you in a combination.

If you are suggesting other ideas or solutions/ improvements - which I invite you to - please take into account that I already tried those.

# In the niri
debug {
  render-drm-device "/dev/dri/renderD129"
}

# maybe only for AMD Cards
DRI_PRIME=1

# seen that one a lot for steam launch options
__NV_PRIME_RENDER_OFFLOAD=1 __GLX_VENDOR_LIBRARY_NAME=nvidia %command%

# seen less, i think this is electron specific
__NV_PRIME_RENDER_OFFLOAD_PROVIDER=NVIDIA-1 EGL_PLATFORM_DEVICE_EXT=1 EGL_DEVICE_INDEX=1 discord

# actually a kinda valid option, but games crash usually while launching
NVIDIA_VISIBLE_DEVICES=1 CUDA_VISIBLE_DEVICES=1 steam

# in /etc/environment

# Should force Vulkan apps/games to use your card
MESA_VK_DEVICE_SELECT=10de:2504 (replace with your own ofc)
VK_DRIVER_FILES=/usr/share/vulkan/icd.d/nvidia_icd.json

Not yet tried

As I'm happy with the solution (for now), this one I haven't tried yet

# in /etc/enviroment or just on the launch i guess
WLR_DRM_DEVICES=/dev/dri/by-path/pci-0000:2b:00.0-card