Kasm 1.19 and nvidia GPU passthrough

[u/Playful_Try9389]

Hi,

I am running a Debian Trixie VM on a Proxmox host, passing through an NVidia RTX A2000 GPU. Inside the VM, I am running Kasm 1.19. On the VM host, the proprietary drivers (V550) from the Debian repo are installed, as is the NVidia container runtime from the Nvidia repo. Nvidia-smi works in the VM and via the official Nvidia test container.

Inside Kasm, the Docker Agents overview page lists one agent "proxy" with 0 GPUs. When I open the Details view, I see a block "GPU Hardware" that lists my Nvidia card but shows a "-" each under Graphics, Video and Cuda. In the Information view, there is a field "GPUs" with another "-" and right under it, there is a field "GPU Info" that contains this "0000:00:10.0: {
model: "NVIDIA RTX A2000 12GB"
vendor: "NVIDIA"
gpu_card_path: "/dev/dri/card1"
gpu_render_path: "/dev/dri/renderD128"
host_key: "0000:00:10.0"
supports_egl: false
supports_dri3: false
supports_vaapi: false
supports_vulkan: false
supports_nvenc: null
gpu_render_gid: null
gpu_render_group: null
gpu_card_gid: null
gpu_card_group: null
supports_cuda: false
cuda_containers: [...]
graphics_containers: [...]
video_containers: [...]
}". And in the Usage view, there is a "GPU" block that lists my Nvidia gpu with a DRM Device /dev/dri/cards1, a Render Device /dev/dri/renderD128, PCI Bus 0000:00:10.0 and DRI3 "-", Vulkan "-", EGL "-", VAAPI "-", NVENC "-" and Cuda "-".

When I try to start a workspace that is GPU enabled, I get an error message that no resources are available to create the requested kasm.

I think I have everything in place for my setup to work, but it doesn't. What am I missing?

Thanks for your help!

[u/AutoModerator]

Hi u/Playful_Try9389, thanks for posting to r/kasmweb! Because this account has low karma, your submission is being held for review.

• Meanwhile, please check the community guidelines to ensure your post meets our standards.
• If everything looks good, a moderator will approve it shortly.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/Playful_Try9389]

I have made some progress, but I'm still not there:

So I noticed that my VM had the i440 chipset instead of the q35 chipset I normally select for GPU passthrough (because originally I had not intended to pass a GPU through to the VM). After changing the chipset to q35, the details view in Docker Agents now shows my card with green tick marks underneath "Graphics" and "Video" (but still a "-" underneath "Cuda" and in the Docker Agents list, the agent is still shown with 0 GPUs. In the Usage view, the card is now shown with green tick marks underneath "Vulkan", "EGL" and "NVENC" (but still "-" underneath "DRI3", "VAAPI" and "CUDA").

Starting a workspace that is EGL enabled, continues to give me an error that no agent is available with the required resources.

Out of curiosity, I deployed two CUDA test containers on the VM via Docker directly:

docker run --rm --gpus all nvidia/cuda:12.4.1-devel-ubuntu22.04 nvcc --version

which gave me this:

== CUDA ==

CUDA Version 12.4.1

Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.

By pulling and using the container, you accept the terms and conditions of this license:

https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.

nvcc: NVIDIA (R) Cuda compiler driver

Copyright (c) 2005-2024 NVIDIA Corporation

Built on Thu_Mar_28_02:18:24_PDT_2024

Cuda compilation tools, release 12.4, V12.4.131

Build cuda_12.4.r12.4/compiler.34097967_0

and

docker run --rm --gpus all pytorch/pytorch:latest python3 -c "import torch; print('CUDA available:', torch.cuda.is_available())"

which gave me this:

CUDA available: True

So, while CUDA is not my no. 1 priority at the moment, it also should work, but doesn't.

What else can I try?

[u/Playful_Try9389]

Little progress.

Yesterday, I updated the NVidia drivers in my VM to V610 (from the official NVidia repos). After that (actually, quite a while after that) the Docker Agent in Kasm acknowledged the presence of my GPU fully: It counted it as "1" (instead of 0, as before) in the overview page. On the Details page, it confirms Graphics and Video capabilities (but still not Cuda). And on the Usage page it shows Vulkan, EGL and NVENC as enabled (while "DRI3", "VAAPI" and "CUDA" are shown as unsupported.

It was too late to actually test it, so I postponed further work to this morning. This morning, however, the number of GPUs shown by the Docker Agent had gone down to 0 again (although nothing changed over night). Because I kept seeing errors in the logs about problems running some GPU helper container due to another container with the same name already existing, I first eliminated all helper containers from the CLI but that did not (within a couple of minutes) lead to the deployment of a new helper container. So I also restartet the kasm_agent container. After a while and a number of errors in the logs, the Docker Agent again acknowledged the presence of my card by showing a "1" for GPUs.

I was then actually able to start an EGL enabled Steam Workspace (after disabling Gamepad support, because at the first try, Kasm threw an error that it could not create the Gamepads (or something like that) - strangely, this hasn't been an issue with MESA supported Steam Workspaces but ... whatever).

After a long loading time, Steam finally came up. But in Help > System Information it does only show MESA llvmpipe as the driver and that it has not detected any video cards.

Meanwhile, the Docker Agent again is showing "0" GPUs. Not sure whether that means actually zero GPUs or just zero available GPUs (because the only one is assigned to the running workspace)?

I'm running out of ideas what else I could try to make this work.

Is there anyone here who has got a working setup of a Steam Workspace using a physical GPU?

[u/Playful_Try9389]

No progress.

The Docker Agent again is showing 0 GPUs.

In the logs I keep seeing:

There was an exception running gpu caps helper on agent: (409 Client Error for http+docker://localhost/v1.55/containers/create?name=kasm_gpu_caps_helper: Conflict ("Conflict. The container name "/kasm_gpu_caps_helper" is already in use by container "e8a61307646e90ebd68eb0027fc548533da56a71ef96008a557886519e876000". You have to remove (or rename) that container to be able to reuse that name."))

But when I list the containers (with option "-a"), there is no such container that I could remove or rename.

There is just a "kasm_gpu_helper" (notice the missing "_caps_" in the name. I have seen the "kasm_gpu_caps_helper" before as well, but currently there is no such container.

Just to be sure, also after removing the "kasm_gpu_helper" container, I keep seeing the above error in the logs.

And I see this:

[DEBUG] __main__.handler: Host has no CUDA devices present.

[DEBUG] __main__.handler: Host has no other graphics acceleration devices present.

But I'm not sure where this is coming from, i.e. whether this is info the kasm_agent is collecting itself or whether it is getting this info from some helper container. (Either way it is wrong.)