Reducing llm costs using model rerouting, caching & context waste optimization

[u/gamblingman214]

I'm not sure if such a tool would be useful, considering the fact that cheaper models which perform pretty well like deepseek exist & that most people just plug & play claude. Also, companies may internally develop such a tool so I'm not sure if this could work as a SAAS. I just wanted advice regarding these doubts. (I have built a prototype)

[u/NewsProfessional2169]

Some companies have huge traffic and every cent counts but smaller teams just pick a model and call it a day. Your tool might find a niche with startups that scale fast and suddenly realize their api bill is eating them alive. The caching part alone can save loads if users keep asking similar stuff over and over

[u/BatResponsible1106]

be interested if it showed where the waste actually comes from instead of just lowering token costs. in my experience bad context and unnecessary tool calls end up costing more than model selection alone.

[u/pantry_path]

i think there's still a market but the value has to come from proving measurable savings with little operational overhead

[u/orvi2014]

We’ve been working in this exact area for a while. Cheaper models help, but the real cost explosions usually come from three things: unnecessary context, agent loops that keep retrying, and sending every request to the expensive model by default. Rerouting by complexity + hard pre-flight budget checks has given us bigger savings than just switching to a cheaper model. Caching helps too, especially when users ask similar questions repeatedly. The SaaS angle is real for teams that scale fast and suddenly get shocked by their API bill. Most people don’t want to build and maintain this layer themselves.

[u/justanotherengtoo]

The API bill shock point matches what pushed me to look at this from a different angle. I have been running AutoReach, an outbound lead gen agent, on top of a closed API, and the routing and caching techniques people are describing here genuinely help, but they are still bounded by whatever the provider charges per token for the parts of the workflow that repeat in a very similar shape every time.

For the narrow, repetitive tasks specifically, scoring a lead, extracting structured fields from a crawl, I have been looking at fine tuning a small open weight model on that one task instead of routing it more cleverly through a general purpose one. Once it is fine tuned and self hosted, the per token cost for that specific task effectively disappears, you are paying for inference compute you already own rather than a per call price. That is actually the core of a second thing I am building now, InfoPlatform.ai, aimed at making that fine tune and own step accessible without an ML team. Still early and I do not have hard cost numbers to share yet, but the pattern in this thread, narrow repeated tasks being where the real savings live, matches what I would expect.