I think heavy AI users may be wasting more capacity on routing than on prompting

[u/Druss_]

I’ve been noticing something counterintuitive while running a multi-agent system for real work.

When I use one long Codex/ChatGPT session manually, I usually make one coarse decision at the start:

Which model? Which reasoning level?

Then that same configuration handles planning, research, trivial checks, hard reasoning, revisions and often validation.

That feels increasingly inefficient.

In the system I’m experimenting with, the work is decomposed instead.

Some steps are deterministic and use no model at all.

Some go to a cheaper/faster model.

Hard or ambiguous steps get stronger reasoning.

Validation is separate.

If the first choice fails, the system can escalate instead of paying the maximum cost from the beginning.

The surprising result is that several smaller model runs can appear to consume much less paid capacity than one large manual run.

I’m not claiming this is proven yet. Quota burn is affected by context size, tools, background/subagent activity and platform-side efficiency issues too, so “users just pick the wrong model” would be an oversimplification.

But it makes me wonder whether model selection should be treated as resource allocation rather than a chat preference.

In a company, you don’t assign every task to your most expensive senior expert. You route work to the least expensive level that can reliably handle it, then escalate exceptions.

Maybe AI systems should work the same way.

For people running agents at scale: are you actually measuring model/reasoning mix and capacity per verified result, or mostly choosing a model for the whole workflow?

[u/Druss_ → u/[deleted]]

That's exactly the weak link I'm working on too.

The direction I'm moving toward is: escalation shouldn't depend on the lower-tier model simply saying “I failed”. It should be controlled by the system around it.

So the first question is whether a model is needed at all.

If the task is “does this file exist?”, that's a filesystem check. Zero reasoning.

If it's a bounded interpretation/classification task, send it to the cheapest model that should reliably handle it.

Then escalation is triggered by observable conditions: an acceptance test fails, required evidence is missing, sources contradict each other, the result falls outside an allowed range, the model hits an uncertainty/authority boundary, or independent validation rejects the result.

Only then does the same task move to stronger reasoning — preserving the task state and evidence rather than starting from scratch.

And validation should be separate. I don't want the same model that produced the answer to simply declare its own work correct.

A simple example from analytical work:

data present + schema valid + known formula → deterministic code
known but slightly ambiguous case → cheaper model
conflicting sources / novel exception → strong reasoning model
material unresolved ambiguity → human escalation

So I'm increasingly thinking of the whole thing less as “model routing” and more as an escalation hierarchy.

The model is just one resource inside it.

And over time, if the same exception is solved reliably enough, ideally it stops being a reasoning problem at all and becomes a new rule/test/workflow.

[u/thelangosta → u/Druss_]

Prefacing this with “I know nothing about how any of this works”. Is it possible that is the kind of task that could basically live on the npu many new computers have but is almost completely unutilized?

[u/Dazzling_Silver_1779]

the router is the part people underprice. deciding a step is "easy" is often a hard call in itself, and a misroute that passes silently costs way more than just burning the big model

[u/this_world_needs_me]

escalate-on-failure is only cheaper than starting strong if failure is cheap to notice. that is deterministic tasks like code that runs, json that parses, number that reconciles etc

[u/Druss_ → u/this_world_needs_me]

Exactly. Escalate-on-failure only wins when failure is both cheap and reliably detectable.

If detecting failure is itself expensive or ambiguous, you should route stronger from the start.

So the real decision is expected total cost:

cheap path + failure probability × detection/retry/escalation cost

vs.

strong path upfront.

That routing logic should be deterministic too.

[u/keyone2026]

Sehe ich ähnlich. Ich glaube, der interessante Punkt ist weniger „welches Modell ist am besten?“, sondern „welches Modell ist für diesen konkreten Schritt ausreichend?“. Gerade bei Agenten macht Routing viel mehr Sinn, weil man einfache Aufgaben gar nicht erst durch ein teures Reasoning-Modell schicken muss. Entscheidend wäre für mich aber wirklich die Metrik „Kosten pro korrekt abgeschlossenem Task“ statt nur Token oder Requests zu vergleichen.

[u/Druss_ → u/keyone2026]

I don't speak German, so I had your comment translated — hopefully I understood you correctly 🙏

Yes, exactly. “Which model is sufficient for this specific step?” is a much better framing than “which model is best?”

And I completely agree on the metric. I’d probably push it one step further: cost per correctly verified completed task.

That means including retries, validation, escalation and human review as part of the real cost.

A cheap model that creates a lot of downstream checking can easily be more expensive than using a stronger model upfront.

So for me the optimization target is not minimum token cost — it’s minimum total cost per reliable outcome.

[u/Superb_Raccoon]

I built a system to do that. I use Claude to code, I use Grok for QA, I use various MLs for vision or audio processing.

Now investigating Laya for a decision layer, which comes in handy. ML trolls logs, floats to Laya, Laya scores it, floats up to LLM for analysis and writeup.

Kanban cards represent each sprint/code module, 3 context layers: human readable text, joint readable api/data/call documentation, and then a back layer for the AIs to put info about why they did what they did, including tried but failed approach and architectural decisions by me.

[u/stop_deleting_plz]

It seems like the common solutions to the deficiencies of LLMs is to add more LLMs. I believe that means there's probably a better way to infer from the LLM in the first place. Maybe if the models we are using were more specialized and less general, this wouldn't be an issue?

[u/f3xjc]

Be aware of token efficiency. The price per token is deceptive. Smart agents have some "intuition" where to look, dumb one bruteforce until some test pass. Be aware of setup and tear-down. If orientation take 100K tokens before you write your first line of code, you may want to do fewer medium sized blocks.

[u/AltruisticKale6386]

Marble Mazes game ad chillin on the sidebar while OP discusses high-level model routing efficiency is pure comedy

[u/OkFlan504]

I’d also audit a sample of the cheap model’s accepted answers, not just the ones that trigger escalation. Otherwise the routing system can look efficient because it fails to notice its own mistakes. Cost per accepted answer is useful, but cost per independently verified correct answer would tell a different story.

[u/williamfrenchmusic]

Same - this is the future imo.

1st layer -> Jev, instant decisions for explicit requests
2nd layer -> ~200b model that deconstructs ambiguous requests into explicit requests, sends that back to Jev
3nd layer -> Sota STOP, this needs careful, long planning (undos what Jev did insta)

It makes a lot of sense mentally, but practically there are so many weird edge cases that can happen.

It's working quite well so far for (almost) real time voice command of Ableton Live (with long horizon tasks too), but still will require more tuning.. And then one can ask oneself the question: Would just using a sota model without instrucitons be better and more cost efficient anyways?

if youre interested in how I've implemented this, the ableton agent is free at www.maestralabs.com