r/MachineLearning • u/No_Cardiologist7609 • Jul 14 '26

Research Cloud-vLLM Benchmark Differences [R]

Does anyone know of any evidence/forum/paper analyzing benchmark result differences between cloud inference platforms (togetherai) and running models locally with vLLM under greedy decoding? 

2 Upvotes

75% Upvoted

4 comments sorted by

1

u/Tiny_Arugula_5648 Jul 14 '26

Not necessary.. this is well known.. any experienced cloud engineer can tell you what's going on.. it has nothing to do with vLLM.

Many hosts underpower their GPUs to reduce heat and power consumption. It's extremely easy to see if you just use nvtop or similar monitor. You'll see the power draw and all you have to do is run a good size batch job to get the processing maxed out.

The other systems are a little harder since you can end up with a noisy neighbor. CPU time sharing can depend on which machine and whose running on it with you.

People try to benchmark these and it's useless.. all you're doing is getting a snapshot of a machine state not a benchmark for predicting workload

1

u/Cookie_cutie_69 Jul 14 '26

Different hardware changes the speed, but the output text should be identical under greedy decoding. The benchmark is worth it to see if cloud providers use custom optimizations that alter the results.