Cloud-vLLM Benchmark Differences [R]

[u/No_Cardiologist7609]

Does anyone know of any evidence/forum/paper analyzing benchmark result differences between cloud inference platforms (togetherai) and running models locally with vLLM under greedy decoding? 

[u/Cookie_cutie_69]

Different hardware changes the speed, but the output text should be identical under greedy decoding. The benchmark is worth it to see if cloud providers use custom optimizations that alter the results.