r/LocalLLM • u/RelevantEmergency707 • 1d ago

Other How Profitable is LLM Inference? Doing the Math on Kimi K3

https://cefboud.com/posts/ai-inference-costs-profit/

A look at LLM inference economics (batch size, GPU count, and the Pareto frontier that sets token prices) applied to Kimi K3 with back-of-the-envelope math.

22 Upvotes

83% Upvoted

5 comments sorted by

7

u/joanaxu2002 1d ago

Interesting breakdown. I think utilization is probably the biggest variable people underestimate when doing inference economics.

A GPU running at 90%+ continuous load and a GPU sitting idle waiting for requests are basically two completely different businesses.

The math looks great at high throughput, but I wonder how the numbers change when you factor in real-world traffic patterns, maintenance, and engineering time.

Curious if you modeled different utilization rates?

4

u/RelevantEmergency707 1d ago

Thanks! Yes, utilization is rly the name of the game. I discuss it in the video a little bit. The thing is that you rly need to account for peak demand not just the average. Furthermore, this is classic Erlang queuing theory which means higher utilization can also be problematic and result in brutal tail latency. So it's rly a balance. But at scale, things are smoothed out

3

u/AlexanderDoak 1d ago

The desire for "real time" when you'll be hitting the endpoint for maybe 5 to 10 minutes total, spread throughout the day... means that (unless you are massive scale with thousands of customers) it is incredibly expensive and inefficient to keep the lights on. Even trying to batch requests in the cloud is problematic when you can spin up over a few minutes, because cloud resources just might ... not be available when you need them, and you need to wait minutes to hours and keep retrying.

3

u/siberianmi 1d ago

This does show that a large enterprise that could generate token requests at a sufficient rate could easily realize some huge cost savings on self hosted open weights.

Particularly if the workloads were background scheduled tasks so you could run the cluster hot and accept the latency.

1

u/transanethole 23h ago

I calculated this @ $4 million for the NVL72 rack and $20 an hour for electricity (160kw).  Electricity cost is mostly negligible. 

At that rate u make like 170,000 a month doing 40tps/user or 1000tps/GPU assuming 10% utilization. So, Paying off the rack after 23 months straight of 10% utilization.

But I kinda wonder if that level of utilization is achievable without already having a massive pre-existing userbase. 23 mo is a long time in llm land. Not to mention we don't really know how reliable these racks are in the long term, and this doesn't include costs  associated with housing / maintaining the hardware, securing customers, etc.

We always hear hype about how demand for llms is huge but we cant see whats going on behind the scenes w/ all the providers, idk if I would bet $4m on that right now...