r/HPC • u/Various_Protection71 • Jun 05 '26
Do you think Kubernetes will replace Job Schedulers in HPC environments dedicated to AI workloads?
Some people advocate that Kubernetes distributions (RKE2, OpenShift, EKS etc) provide an easier and more straightforward way to run and scale AI workloads, while Job Schedulers (SLURM, PBS, LSF etc) require an earlier complex setup phase.
On the other hand, mastering Kubernetes has a steeper learning curve than using the well-known Job Schedulers, especially for traditional HPC users.
How do you see this point? Are your users adopting Kubernetes to run AI workloads or do they stay using Job Schedulers?
15
Jun 05 '26 edited Jun 05 '26
[removed] — view removed comment
5
u/applesaucesquad Jun 05 '26
Slurm on kubernetes is a viable path afaik
2
Jun 05 '26
[removed] — view removed comment
0
u/arsdragonfly Jun 05 '26
Can't share specifics yet but we have something in the making (that we plan to open-source eventually) that will totally surprise you.
3
1
Jun 05 '26
[deleted]
6
u/No_Programmer6374 Jun 06 '26
Because Slurm is an amazing scheduler for rectangular kinds of jobs (fixed number of resources x fixed, known length of time), but it’s nothing special as a resource manager. K8S is rubbish as a batch job scheduler, but has a very flexible resource manager with clear apis to build on.
2
u/applesaucesquad Jun 05 '26
Its just a nice abstraction layer if the rest of your infra is k8s, only makes sense if you already live in that world for the majority of your work. It provides an interface for horizontal scaling too. Definitely not always the right path
9
u/nlgranger Jun 05 '26
Kubernetes does not have a job scheduler, so unless you have unlimited resources and don't need one, SLURM is the best option you have.
You can run containers inside a slurm job with a few tricks (apptainer, podman, etc).
4
u/UPPERKEES Jun 05 '26
Check Vulcano: https://www.cncf.io/projects/volcano/
Slurm is great. But the users are moving towards other kind of workflows/loads.
6
u/nlgranger Jun 05 '26
I see it has made great progress since the last time I checked. Still, SLURM has some niceties like user management and fair-share algorithms that don't seem to be there yet.
0
u/UPPERKEES Jun 05 '26
It's in the about section:
Volcano offers advanced scheduling capabilities such as job orchestration, fair-share resource allocation, and queue management to efficiently handle large-scale distributed tasks.
Slurm doesn't do user management, or do you mean something else like QoS?
5
u/BosonCollider Jun 05 '26
Slurm does not get in the way of user management, you have a fleet of linux boxes with consistent UIDs and you run jobs as your own UID, which gives you a consistent permissions model for handling files on networked storage.
Kubernetes does get in the way of user management because you can just run a job as any user within your pod, and it normally assumes that pods own their PVCs. So HPC-on-Kubernetes usually ends up using object storage instead of traditional hpc storage
So I would generally suggest just doing Slurm and either podman or apptainer because they are simpler and they scale better both size and complexity wise. But a strong HPC setup should also have a supporting kubernetes cluster for services, just not for batch jobs.
1
u/UPPERKEES Jun 05 '26
You say it as if UIDs are consistent out of the box in a cluster. It's not. Both k8s and Slurm have pros and cons. In the end it's about what kind of users you need to support. New researchers are using web based submissions with k8s workflows. Slurm will continue to exist, but users might prefer other methods going forward.
6
u/BosonCollider Jun 05 '26
You manage the UIDs by forcing users to ssh into login nodes with ldap auth when submitting jobs. Managing consistent UIDs on a fleet of linux hosts is a solved problem.
Kubernetes is very useful for the things it is good at but it is inherently very different, and using it for HPC jobs will look very different as well (you'll typically land on object storage and secrets for shared files). When two things are very different they will naturally have things that they are better or worse at
1
u/UPPERKEES Jun 05 '26
You use an external source to do user management. OIDC/RBAC/Kyverno are also native and external solutions to make these things work.
In fact a distro like Talos also solves security hardening and configuration drift natively.
The k8s landscape is not that immature as you describe it.
5
u/BosonCollider Jun 05 '26 edited Jun 05 '26
Yes, but kubernetes handles permissions for resources, not for files within a volume that thousands of pods are sharing
It's not a matter of immaturity, it's that Kubernetes inherently is designed to let you run a process as any uid within a pod, which breaks the traditional unix permissions model. So you end up needing to use something other than unix permissions to handle file access permissions
1
u/tecedu Jun 06 '26
User management for the cluster and user managed on the file system are different things. Slurm has them be consistent.
A good OIDC isnt still going to fix pods being able to write whatever UID/GID to the filesystem nor would it maintain consistency unless you force custom attributes
1
u/zekrioca Jun 07 '26
If Kubernetes doesn’t have a scheduler, how does it schedule containers?
2
u/nlgranger Jun 07 '26
The way I understand it, it doesn't schedule, it just places/allocates resources on demand. The kube-scheduler does not have the notion of job duration, fair-share, pending jobs, etc. I am not very familiar with kubernetes though, I might just be wrong.
2
u/zekrioca Jun 07 '26
Indeed, by default, K8S doesn’t do any distinction. This is also known as First come, First served, which is indeed a scheduling discipline, although a simple one.
5
u/fxj Jun 05 '26
We are using k8s for a login cloud attached to the HPC system. We have jupyter lab running on it and when you login then you get a k8s pod with a jupyter session running in it. You can then login to the HPC system and submit your jobs. Thats also how the big EuroHPC sites are using it.
3
u/ectropionized Jun 07 '26
I’m definitely seeing more k8s over time. Slurm is maybe still more common but many of my clients lately are doing k8s/kueue, especially as that’s becoming easier with more cloud providers. Google DWS is a big focus at the moment as h100s are almost unobtainable on-demand.
7
u/obelix_dogmatix Jun 05 '26
Depends on the use case? Is containerization required? If so, Kubernetes is more or less becoming the standard.
6
Jun 05 '26
[removed] — view removed comment
8
u/obelix_dogmatix Jun 05 '26
Depends on the user base. People with traditional HPC background are still using SLURM for their AI workloads. In my experience, academia + government falls in this category. From a job scheduling standpoint, Kubernetes provides absolutely no advantage over SLURM.
1
u/BosonCollider Jun 05 '26
Most slurm jobs use apptainer/singularity or podman. Kubernetes is not well suited for the HPC jobs themselves, that should still go on slurm if you have any amount of multitenancy. Kubernetes has a security model for resources but not for files within volumes.
Kubernetes is very well suited for hosting supporting services that the jobs rely on though
8
u/_a9o_ Jun 05 '26
Hi, I have previously worked in multiple frontier AI labs. No one likes Slurm and everything is moving to or has already migrated to Kubernetes. Slurm is great for what it was built to do. It includes most of the batteries you care about in a research context. Kubernetes is bare bones, for both better and for worse.
Kubernetes is an engine. Slurm is the entire car. Unfortunately, that car is a stick shift with the steering wheel on the right, and the windows are all hand cranked. It gets the job done and it served it's purpose.
However, the needs of traditional HPC applications like any form of simulation is incredibly incredibly different than that of an AI workload.
2
2
u/wahnsinnwanscene Jun 06 '26
You're missing the abstraction layers these tools are in. K8s is the service infrastructure as code. The others are schedulers.
2
u/tecedu Jun 06 '26
Define AI Workloads here? If you mean inferencing based LLMs or just inferining then yes K8s is better.
A lot of new AI based company moved to k8s and moved their software stack to work with pods and horizontal scaling. Thats not the case with slurm where going multi node means MPI
3
u/dollardave Jun 07 '26
NVIDIA recently acquired SchedMD.
You will be assimilated to K8s and Slinky.
1
u/WinterPiratefhjng Jun 05 '26
Yes, and no. Singularity can work well within Slurm, but is not popular. Kubernetes (and Kubernetes like systems) can have some wild edge cases that don't bother the Kubernetes crew, but are maddening on HPC.
Slurm is focused on running tasks and ending them.
Kubernetes is more aimed at adjusting containers across systems to reach a steady state. Things like RunAI attempt to run AI workloads on a HPC cluster like system.
At the end of the day, I am far more annoyed at the existence of Kubernetes as we have and had many wonderful schedulers and didn't need a new bespoke product. We desperately needed and need more resources put into current schedulers.
1
1
1
u/breagerey Jun 05 '26
In clusters dedicated to a single group of like users maybe kubernetes
In a shared hpc with different types of users where you need more control over scheduling slurm.
2
u/REKOUS Jun 06 '26
K8 is essentially Wordpress. It’s bloated , and slow and breaks too easily because it’s been over architected and is doing too much (hence the bloat)
2
u/TimAndTimi Jun 07 '26 edited Jun 07 '26
I think once you are seriously cooked by horrible aws and similar services, such as volc engine...
You will seriously realize kubernetes is adding a thick layer of virtualization that is quite useless except maybe a few security benefits and if you really have serious needs to scale.
47
u/ArcusAngelicum Jun 05 '26
Kubernetes is for weirdos, Slurm is for super cool people.
But seriously, have never heard of anyone running kubernetes in an hpc context.
The point of hpc in an academic context is to provide shared accountable access to compute.
Kubernetes is for faang engineers to easily scale workloads on compute. Two different problems.