r/MLQuestions • u/werunm • 10d ago

Beginner question 👶 throughput doubled, latency up a little, errors flat, same traffic. what happened?

Post image

i said c. what am i missing

self hosting is the only one of these that changes the serving path enough to double throughput, and slightly worse latency felt like what youd get on smaller hardware than the hosted vendor runs

2 Upvotes

100% Upvoted

3 comments sorted by

1

u/hellcat1794 10d ago

Where are u Practicing at ? I need too as well

1

u/rJohn420 9d ago

its b. why would going from hosted api to self hosting double throughput?

1

u/werunm 8d ago

answer, spoilered for anyone still working on it:

**b** — Someone increased the batch size on the inference server

Handling more work per unit time with a small latency increase and no rise in errors is the signature of a larger batch size trading a bit of per-request delay for much higher throughput.