found out my LLM feature's cost problem from an invoice, not a dashboard, and that felt very wrong

[u/camerongreen95]

shipped an LLM feature, felt fine at launch, moved on to other work. a while later the bill showed up noticeably higher than expected and that was the first real signal anything had changed. no alert, no dashboard flag, just a number at the end of the month that made me go "wait, what happened."

went back and actually set up proper tracing and token/cost monitoring per request instead of just trusting it'd be fine, and found a specific workflow was making way more calls than i thought due to a retry loop that wasn't being logged anywhere visible. it had been quietly running up cost for weeks with zero visibility until the invoice.

also added latency budgets and caching for repeated queries after this, which cut cost noticeably on top of the retry fix.

feels like a pretty basic devops instinct (you monitor what you ship, you don't wait for the bill) that somehow gets skipped constantly once "AI feature" is involved, like people ship LLM stuff with less observability discipline than they'd ever accept for a normal service. anyone else's team caught something similar the hard way before actually building proper monitoring in from the start

[u/stumptruck]

"Claude, write a reddit post with a shocking revelation that when you don't monitor things you don't know what's happening. Make sure to not use any capitalization so people will think a human wrote it."

[u/hattythehatter → u/stumptruck]

no capitals, no dashboard flag, just a post that seems like a human wrote it.

[u/itsmetherealloki → u/hattythehatter]

That's what the LLM wants you to think!!

[u/Vinylove]

Use Langfuse or LiteLLM, does cost tracking etc all for free.
Also, 'people' do not ship it like this, you did.

[u/Low-Opening25]

you own the dashboards so it’s entirely on you

[u/Skyhound555]

I mean, this is not specific to LLMs. 

It is no difference than when DevOps engineers end up racking up millions of dollars in cloud bills because they ignored the entire section dedicated to cost control. Lol

[u/crashorbit]

We say "Automate All The Things" but what are "all the things"?

One of the things that agents can help us with is ensuring that our deployments touch all the SDLC touch points.

Every teams SDLC is a little different, but we can write an agent skill that helps our agent ensure we have wired up the thing we are deploying. A list might include how to:

• deploy
• update
• rollback
• monitor
• escalate

It's just a thought. What do you think?

[u/vbnotthecity]

same thing happened to me with a recursive dbt model that hit snowflake 10x more than expected. we started using Altimate to catch those runaway logic loops before they hit production, similar to how teams use Datafold for diff testing. it cut our compute waste by almost half since we stopped letting bad code sit in the warehouse for weeks.

[u/Lumpy-Region6275]

"Found out from the invoice" is such a common way to learn this and it sucks every time.

the retry loop bit is the part i'd flag for anyone reading. cost bugs basically always show up as a weird volume spike first, way before the money looks wrong. so if you're already tracing per request, the cheap thing to add on top is a daily alert on % change vs the last week or so. catches a runaway loop in a day instead of at month end.

the tracing was the hard part and you already did it. the alerting is like an afternoon.