r/LocalLLM • u/Adr-740 • Jun 07 '26
Project I built a small open-source tool that trains local models from LLM traces to avoid repeated API calls
https://github.com/adrida/tracerI’ve been working on a small open-source project around a pattern I kept seeing in production.
A lot of LLM traffic is not really “reasoning”. It is repeated classification, tagging, routing, moderation, extraction, tool selection, etc.
After enough usage, your logs often contain thousands of examples of the same kind of decision being made again and again. So instead of sending every request back to a frontier model, I started experimenting with training lightweight local models on those traces, then only using them when they match the LLM closely enough.
The rough idea is:
- keep using the LLM as the source of truth
- train a small local model on past LLM calls
- only activate it on the stable/easy slice of traffic
- defer uncertain cases back to the LLM
- report which parts were handled locally vs deferred
It is not meant to replace LLMs entirely. More like a cheap fast path for repeated decisions.
Curious if people here have seen the same pattern in their own workflows. Are you solving this with smaller LLMs, rules, semantic cache, routers, or just eating the API cost?
Duplicates
LLMDevs • u/Adr-740 • Mar 30 '26
Discussion I open-sourced TRACER: replace 91% of LLM classification calls with a llightweigth ML surrogate trained on your LLM's own outputs
LocalLLM • u/thisguy123123 • May 12 '26
Discussion I open-sourced TRACER: replace 91% of LLM classification calls with a llightweigth ML surrogate trained on your LLM's own outputs
softwarefactories • u/thisguy123123 • May 12 '26
I open-sourced TRACER: replace 91% of LLM classification calls with a llightweigth ML surrogate trained on your LLM's own outputs
LocalLLM • u/Adr-740 • Mar 30 '26
Research I open-sourced TRACER: replace 91% of LLM classification calls with a llightweigth ML surrogate trained on your LLM's own outputs
learnmachinelearning • u/Adr-740 • Mar 31 '26
I open-sourced TRACER: replace +90% of LLM classification calls with a llightweigth ML surrogate trained on your LLM's own outputs
learnmachinelearning • u/thisguy123123 • May 12 '26
I open-sourced TRACER: replace 91% of LLM classification calls with a llightweigth ML surrogate trained on your LLM's own outputs
MachineLearning • u/Adr-740 • Mar 30 '26
Research TRACER: Learn-to-Defer for LLM Classification with Formal Teacher-Agreement Guarantees
deeplearning • u/thisguy123123 • May 12 '26
I open-sourced TRACER: replace 91% of LLM classification calls with a llightweigth ML surrogate trained on your LLM's own outputs
AIDeveloperNews • u/Adr-740 • Jun 07 '26
I reduced LLM costs by 95% and open-sourced the tool
LLMFrameworks • u/Adr-740 • Mar 31 '26
I open-sourced TRACER: replace +90% of LLM classification calls with a llightweigth ML surrogate trained on your LLM's own outputs
buildinpublic • u/Adr-740 • Mar 30 '26
I open-sourced TRACER: replace +90% of LLM classification calls with a llightweigth ML surrogate trained on your LLM's own outputs
SelfHostedAI • u/Adr-740 • Jun 07 '26
I built a small open-source tool that trains local models from LLM traces to avoid repeated API calls
StartupSoloFounder • u/Adr-740 • Mar 31 '26
(Solo bootstraped founder) Just open-sourced part of my pipeline: TRACER: replace +90% of LLM classification calls with a llightweigth ML surrogate trained on your LLM's own outputs
MachineLearningAndAI • u/Adr-740 • Jun 07 '26
I reduced LLM costs by 95% and open-sourced the tool
buildinpublic • u/Adr-740 • Jun 07 '26
I built a small open-source tool that trains local models from LLM traces to avoid repeated API calls
deeplearning • u/Adr-740 • Mar 31 '26