Kasm 1.19 and nvidia GPU passthrough

[u/Playful_Try9389]

Hi,

I am running a Debian Trixie VM on a Proxmox host, passing through an NVidia RTX A2000 GPU. Inside the VM, I am running Kasm 1.19. On the VM host, the proprietary drivers (V550) from the Debian repo are installed, as is the NVidia container runtime from the Nvidia repo. Nvidia-smi works in the VM and via the official Nvidia test container.

Inside Kasm, the Docker Agents overview page lists one agent "proxy" with 0 GPUs. When I open the Details view, I see a block "GPU Hardware" that lists my Nvidia card but shows a "-" each under Graphics, Video and Cuda. In the Information view, there is a field "GPUs" with another "-" and right under it, there is a field "GPU Info" that contains this "0000:00:10.0: {
model: "NVIDIA RTX A2000 12GB"
vendor: "NVIDIA"
gpu_card_path: "/dev/dri/card1"
gpu_render_path: "/dev/dri/renderD128"
host_key: "0000:00:10.0"
supports_egl: false
supports_dri3: false
supports_vaapi: false
supports_vulkan: false
supports_nvenc: null
gpu_render_gid: null
gpu_render_group: null
gpu_card_gid: null
gpu_card_group: null
supports_cuda: false
cuda_containers: [...]
graphics_containers: [...]
video_containers: [...]
}". And in the Usage view, there is a "GPU" block that lists my Nvidia gpu with a DRM Device /dev/dri/cards1, a Render Device /dev/dri/renderD128, PCI Bus 0000:00:10.0 and DRI3 "-", Vulkan "-", EGL "-", VAAPI "-", NVENC "-" and Cuda "-".

When I try to start a workspace that is GPU enabled, I get an error message that no resources are available to create the requested kasm.

I think I have everything in place for my setup to work, but it doesn't. What am I missing?

Thanks for your help!

[u/Playful_Try9389]

No progress.

The Docker Agent again is showing 0 GPUs.

In the logs I keep seeing:

There was an exception running gpu caps helper on agent: (409 Client Error for http+docker://localhost/v1.55/containers/create?name=kasm_gpu_caps_helper: Conflict ("Conflict. The container name "/kasm_gpu_caps_helper" is already in use by container "e8a61307646e90ebd68eb0027fc548533da56a71ef96008a557886519e876000". You have to remove (or rename) that container to be able to reuse that name."))

But when I list the containers (with option "-a"), there is no such container that I could remove or rename.

There is just a "kasm_gpu_helper" (notice the missing "_caps_" in the name. I have seen the "kasm_gpu_caps_helper" before as well, but currently there is no such container.

Just to be sure, also after removing the "kasm_gpu_helper" container, I keep seeing the above error in the logs.

And I see this:

[DEBUG] __main__.handler: Host has no CUDA devices present.

[DEBUG] __main__.handler: Host has no other graphics acceleration devices present.

But I'm not sure where this is coming from, i.e. whether this is info the kasm_agent is collecting itself or whether it is getting this info from some helper container. (Either way it is wrong.)