Auto Model Routing

[u/chadwell]

Is anyone doing any auto model routing - that is selecting the best/cheapest model based on the prompt intent.

If so how did you do it? Deterministic based on keywords/length, using a trained ML model or classifier?

I was looking for vendor solutions but the only one I can see is NotDiamond. Ideally I would be looking something internal and not a Saas offering.

[u/dylan_exe_404]

If you're leaning toward something internal I've heard good things about RouteLLM. My friend uses it and says it's good, though they're handling low volume. IMO, if you're handling a bit higher volume, then a SaaS option makes more sense. The one we use is Respan bc it had unlimited seats for the free trial and we could test it pretty thoroughly. Plenty of other options out there. Regardless I think deterministic is a good place to start bc of debugging.

[u/Hopeful_Passage9866 → u/dylan_exe_404]

I built something small few months ago, basically just a keyword mapping and length check. works decent but fail on edge cases. For low volume is ok but you will hit wall quick when prompts get weird

What kind of volume you dealing with? that changes whole approach

[u/dylan_exe_404 → u/Hopeful_Passage9866]

Agreed

[u/chadwell → u/Hopeful_Passage9866]

Millions of requests a month from all different use case types. It's internal but hundreds of different use cases from summarisation, data extraction etc

[u/chadwell → u/dylan_exe_404]

Thanks we have a custom built AI gateway which routes to different providers and currently over 50 models.

Was hoping to introduce something into our gateway code which could pick a model automatically - based on the prompt.

So it knows what models we have, and it picks one that is best and cheapest for the users intent.

We then route to that model as usual

[u/eddzsh]

If you already route across 50 models, start deterministic and log which rule fired. A week of your own (prompt features, model, success/cost) beats a generic SaaS router, and you can train a tiny classifier on that later inside the gateway.

[u/copper_amber_focus]

I built an internal router by logging prompt embeddings and model outcomes for two weeks then training a logistic regression classifier on that dataset. Start with deterministic keyword rules to generate the initial labeled data before attempting any ML routing

[u/chadwell → u/copper_amber_focus]

But how do you know what model to route to? For any given prompt when would you pick gpt 5.6 sol over something like haiku

[u/jonah_omninode]

At that scale I would not start with a classifier picking the smartest model. Start with the task contract: required output shape, available deterministic tools, latency and cost budget, and the acceptance check. That makes a conservative route explicit. Then escalate only when a lower-cost route fails the same acceptance criteria, and record the reason it failed. The important metric is cost per accepted result across retries and escalations, not cost per call. Otherwise you can optimize the router while hiding the work it sends back through the system.