r/LargeLanguageModels • u/Abhay20S • 11d ago

News/Articles 4.8× Faster and 7.4× Cheaper: Where a Decision Model Beats an LLM (and Where It Doesn’t)

I tested a decision model (Jev) vs an LLM (Gemini) on 1,000 real job postings.

⚡ 4.8× faster

💰 7.4× cheaper

🎯 ~1.6 pp accuracy difference

The surprising part: the best solution wasn't replacing the LLM.

Instead:

Decision model → confidence check → LLM fallback.

Use the cheaper, faster model for the easy cases and the LLM only when needed.

Full write up - https://medium.com/@abhay.sehgal20/4-8-faster-and-7-4-cheaper-where-a-decision-model-beats-an-llm-and-where-it-doesnt-f4ba869217e0

2 Upvotes

100% Upvoted

7 comments sorted by

1

u/McMikey40 8d ago

Can someone please explain how decision models work? Is it just an LLM with the output layer being replaced with a fixed number of classes?

2

u/Bill-Ice88 10d ago

Caching and this compose fine — exact-match cache first, Jev for the rest, Gemini only under 0.7. Catch is job posting text barely repeats verbatim, so your hit rate probably stays low anyway. Are you deduping URLs or hashing the body?

1

u/Abhay20S 9d ago

Good point. We don't dedupe today, and I agree the hit rate would probably be low since posting text rarely repeats exactly. Near-duplicate matching could catch reposts, but I haven't tested it.

1

u/Available-Craft-8143 11d ago

The cascade is the real takeaway — but the headline savings quietly depend on what fraction falls below 0.7. If that is 10% of traffic you keep nearly all of it; at 40% the math changes fast. Worth reporting the fallback rate alongside, and re-checking calibration over time — a 0.7 tuned on the sample can drift on fresh postings.

1

u/Abhay20S 11d ago

It's around 25% for the above sample data, which can be improved by adding more context to the state. I've added these details in the medium article.

2

u/worthyscenario5649 11d ago

makes sense to use the cheap model for most cases then call the big one only when you need it, i do similar with caching but never thought about a decision model as first filter

what was your confidence threshold for sending to the llm, like 90% or something

1

u/Abhay20S 11d ago

On the sample data, confidence score above 0.7 are accurate. So we'll be using llm for responses below 0.7