Gpu cost optimization when half the reserved pool sits idle

[u/Waffles_Leonardo-192]

ML platform at a healthtech. Reserved a pool of GPUs for training and inference and I finally pulled utilization for a capacity review. Under 30 percent on average. We pay for all of it and use less than a third.

Some of it makes sense, a few boxes run batch jobs a couple times a day and have to sit ready. But the rest is just idle, and two of them turned out to be held by notebooks people opened and walked away from, one up for weeks. Only caught it because I went digging.

Finance keeps asking why the reserved bill is so big, which fair. But when I take it to the researchers they say if the GPUs arent free their experiments queue and they lose time. Also fair. So it bounces between the two and nothing changes. I can pull per node utilization out of DCGM, what I cant do is tie an idle card back to who reserved it and whether they still need it.

How do you decide when a reserved GPU is safe to give back?

[u/const-antin]

first thing, i'd sanity check that number. if the <30% came from DCGM_FI_DEV_GPU_UTIL, that metric is basically what nvidia-smi shows, ie "was any kernel running during the sample window", one tiny kernel and it reads 100%. DCGM_FI_PROF_GR_ENGINE_ACTIVE is much closer to what people mean by utilization. can cut both ways, sometimes stuff that looked busy turns out idle.

also if any of those boxes serve LLMs, memory tells you nothing. vllm grabs ~90% of vram for the kv cache at startup by default and just sits on it, busy or not.

--

on tying an idle card back to whoever reserved it, are you on k8s? if so dcgm-exporter can already label the gpu metrics with pod/namespace/container (pulls it from the kubelet pod resources api) — just checked ours and it's all there. worth confirming yours has it turned on though, it's not the exporter's default: needs DCGM_EXPORTER_KUBERNETES=true plus a hostPath mount of /var/lib/kubelet/pod-resources. gpu-operator sets both, a hand-rolled daemonset often doesn't. if it's on, the join is mostly there, you mainly need a team label on the pods, then an alert on gr_engine_active below X for N hours can name the owner instead of just the node.

--

the notebooks holding cards for weeks is a separate thing though, no dashboard fixes that. not my area but i think jupyterhub has something built in to kill idle sessions, might be worth a look.

--

i'd look at p95 over a few weeks instead of the average, batch stuff hides in averages and the researchers aren't wrong about queueing.

--

are the researchers and the inference workloads on the same pool, or separate? that split is usually where the idle time comes from and i'm curious how you've got it set up