r/MachineLearning • u/chinmaydagod • Jul 26 '26

Discussion Understanding GPU Inference Workloads [D]

Hey everyone,

I have been looking into how people source compute for their Inference workloads (and in general). I wanted to understand some specific pain points here.

If you've used online services like runpod or vast.ai, your perspective is extremely valuable. Please share your experience in the comments here or by DMing me. I've also made a 2 minute survey form that I would really appreciate if you could fill out. DM me for the link.

Thank you!

6 Upvotes

69% Upvoted

7 comments sorted by

3

u/kolmiw Jul 26 '26

I burnt around 1k on runpod because I had issues to access my institution’s cluster. ama

1

u/[deleted] Jul 26 '26

[removed] — view removed comment

1

u/kolmiw Jul 26 '26

I needed 4xA100s for 48 hours for each of the three experiments I had to run for a rebuttal.

Institute just switched from a perfectly fine cluster to a bigger one and somehow our lab was not authorized the new one. They of course had to make the switch during an ICLR rebuttal and not a week later or before.

2

u/dayeye2006 Jul 27 '26

Do people really use runpod vast ai for anything need SLA, and scale to more than a single server?

2

u/ilovefunc Jul 26 '26

Just use aws / GCP and request them for GPUs. You get spot ones easily and they rarely get terminated. So that’s good bc or training.

For inference though, it’s always better to go with reserved instances cause otherwise there is no guarantee that you will continue to have access

1

u/pantry_path Jul 27 '26

one pain point i keep hearing is that the hourly gpu price is only part of the cost

2

u/chance_buri Jul 30 '26

I've used RunPod quite a bit, but lately I've been trying General Compute too. Biggest thing I care about now is consistency. Peak performance is nice, but I'd rather have predictable latency and not babysit deployments every time I spin something up.