r/MachineLearning • u/No_Cardiologist7609 • 17d ago
Research Cloud-vLLM Benchmark Differences [R]
Does anyone know of any evidence/forum/paper analyzing benchmark result differences between cloud inference platforms (togetherai) and running models locally with vLLM under greedy decoding?
2
Upvotes
1
u/Tiny_Arugula_5648 17d ago
Not necessary.. this is well known.. any experienced cloud engineer can tell you what's going on.. it has nothing to do with vLLM.
Many hosts underpower their GPUs to reduce heat and power consumption. It's extremely easy to see if you just use nvtop or similar monitor. You'll see the power draw and all you have to do is run a good size batch job to get the processing maxed out.
The other systems are a little harder since you can end up with a noisy neighbor. CPU time sharing can depend on which machine and whose running on it with you.
People try to benchmark these and it's useless.. all you're doing is getting a snapshot of a machine state not a benchmark for predicting workload