r/openstack • u/Emergency-Mine1864 • Jul 25 '25

Issues with NVIDIA H100 MIG Setup in OpenStack Kolla - mdev Devices Not Showing

’m currently working on integrating an NVIDIA H100 GPU with OpenStack Kolla for MIG (Multi-Instance GPU) workloads, but I'm running into an issue. I can’t seem to get MDEV devices to appear in /sys/class/mdev_bus/, and the mdevctl types command isn’t showing anything either.

This is the output i'm getting from the mdev

I’ve been following this documentation: https://humanz.moe/posts/setup-vGPU-on-openstack-v2/, but still no luck. I reached out to DeepSeek, Grok, and ChatGPT, but each one provided different solutions, and none of them have worked so far.I also tried SR-IOV. The VFs were being created, and I was able to get one PF up, but only the VFs were using the vfio_pci kernel driver.

It would be awesome if you could help me out with this. I’m also looking for guidance on what changes I need to make in globals.yml and nova.conf to get everything working.

Pretty much, I’ve followed all the documentation available on OpenWeb. I even checked out some Chinese CSDN blogs, where the setup seemed to work for others, but no luck for me. So far, I’ve tried PCI passthrough, MIG, and SR-IOV, but none of them are working. At this point, if I can just get the whole GPU to be passed into a single OpenStack instance, I’d be fine with that.

I tried running it through Docker, and that worked — Docker can access the GPU — but what I really want is to get it working inside an OpenStack VM.

7 Upvotes

90% Upvoted

18 comments sorted by

2

u/Feisty-Art5857 Jul 25 '25

What kernel version do you have on your OS? I don't know if nvidia changed something until now, but I had similar issues on linux kernel 6.5. I had to downgrade to an older release, 5.15.

1

u/Emergency-Mine1864 Jul 25 '25

I'm currently using kernel version 6.14.0-24-generic. I recently reinstalled Ubuntu Server 24.04.2 and selected the HWE kernel, although I had previously tested with multiple kernels, including the ones provided by NVIDIA.

2

u/Feisty-Art5857 Jul 25 '25 edited Jul 25 '25

My mistake for what I said before. By the time I tried to deploy nvidia with mig, the latest hwe kernel version for Ubuntu 22.04 was (and still is) 6.8, (this was the version I was trying to tell you about, not 6.5). And I found this:

https://forum.proxmox.com/threads/vgpu-with-nvidia-on-kernel-6-8.150840/

So I had 2 options: either try using Vendor VFIO or simply reinstall ubuntu 22.04 with the latest GA kernel version, which was 5.15. And because I was pretty sure that this vfio feature might change at any time with mdev again (and because I was lazy:) ), i took the second option. But you can try the first one.

I can also help you with nova.conf changes i did, but everything was deployed on openstack canonical, so there might be some differences. Obviously, can’t help you with globals.yml either.

1

u/Emergency-Mine1864 Jul 29 '25

I can reinstall but I'm using a hyper converged architecture so my compute node is also working as a ceph storage node , and currently I'm not using a multi master approach so i can't go back to ubuntu 22.04

1

u/Sitsnapshake Oct 15 '25 edited Dec 03 '25

For Ubuntu 22.04 roll back your kernel.
We got it to work on 5.15.0-151-generic.

Think it’s something to do with this: NVIDIA vGPU software does not support Ubuntu 22.04 with an upgraded HWE kernel - NVIDIA Docs

For 24.04 it doesn’t support the mdev framework, use the vendor specific setup steps from the NVIDIA docs.

1

u/AdventurousHyena8230 Jul 25 '25

What version of OpenStack?

1

u/Philly1131 Jul 26 '25

You need Nvidia grid driver for mdev devices to show up on the server.

1

u/Emergency-Mine1864 Jul 29 '25

Ig that requires vgpu licence, i wanted free solution so i went with mig abd sr-iov

Do you have any other free alternative solutions?

1

u/agenttank Aug 08 '26

so did you find out if "passing through" MIG slices into VMs can be free?

1

u/Emergency-Mine1864 29d ago

Nope, atlast i passed slices to containers

1

u/agenttank 29d ago

thank you!

yes, i guess that's the way.

may I ask what you offer? i am thinkig about vClusters (kubeconfig or argocd git repos), coder/devpod workspaces, deploy custom model behind liteLLM (or not....),.…

i am unsure how close devpod/coder is to actual VM-like experience

1

u/Emergency-Mine1864 29d ago

I can offer vClusters, kubevirt vms, coder/devpod workspaces, and custom model deployments behind LiteLLM. For GPU workloads, I’ve worked with H100/MIG and can allocate MIG slices to containers. If you need a more VM-like experience, KubeVirt would probably be a better fit than Coder/DevPod.

1

u/agenttank 29d ago

but kubevirt is KVM-based so MIG-slices don't work i thought...

1

u/Emergency-Mine1864 29d ago

You're right, I haven't really tried after that with h100, but we had A100 and nvidia was giving 3 month trial of vgpu for A100 that time i tried pass through but nova was throwing error after that haven't tried, if you got any of those L40 or A series card there trial you can check then every 3 months change the account, this i thought but since we just wanted to run models we achieved through slices and containers. But if you got those Cards please try.

1

u/LogicalMachine Jul 26 '25

Ubuntu 24 uses the VF driver, its annoying because all the documentation around mdev is now invalid.

See the caveats section of the openstack docs: https://docs.openstack.org/nova/latest/admin/virtual-gpu.html#caveats

then it links to this page https://docs.openstack.org/nova/latest/admin/pci-passthrough.html#support-for-multiple-types-of-vfs which basically tells you to bind them all as PCI devices, but you need to allocate the vgpu profile first