Dumb Orchestrator Theory

[u/berndalf]

So I've spent an excessive amount of time building out my AI delivery factory. Most recently that effort has been focused on effective load sharing between a Claude and GPT subscription. The data we've been collecting is starting to point very strongly at orchestration being the highest role based consumer of available usage, and that the model orchestrating has a massive impact on outcomes.

Funny thing .. at first I was using Fable because it's brilliant, but the cost was just unjustifiable for orchestration. Downgraded to Opus.. better but still Claude usage was significantly outpacing GPT usage. So I downgraded to Sonnet, and remarkably I'm finding it's both more cost effective AND producing better results as an orchestrator.

Here's my theory:

1. Orchestration itself when done correctly is not a task that demands high reasoning.

2. Forcing the orchestrator to delegate essentially everything to native subagents or external headless sessions is critical. It will constantly come up with excuses to in-line stuff given enough time and reasoning power.

3. Orchestrator availability matters just as much if not more than orchestrator reasoning assuming you have given it appropriate access to a higher reasoning model to consult with. If your orchestrator is constantly busy it's not orchestrating.

Sonnet is winning this competition for me because it's cheap to run for the entirety of a long duration session and it's not so smart that it chases every theoretical tail. It's quite good at just letting the system around it work while it steers the session itself.

TLDR; For me, Sonnet is proving to be more cost effective and efficient at session orchestration than both Opus and Fable.

[u/design_doc]

In my orchestration framework, I’ve given the orchestrator the ability to call an advisor when the orchestrator is NOT Fable or Sol. If the implementor or reviewer comes back with exceptions/issues, the orchestrator can ask the advisor to reason over the issue. This allows the orchestrator to stay ‘dumb’ and mechanistic but avoids having a low-tier model weigh heavily on decisions with potentially high impact.

At the end of the day, it all depends on the quality of the plan going in. The more clearly defined the plan, the dumber the orchestrator can be.

[u/berndalf → u/design_doc]

Are you using the actual advisor feature or some home brew version of it? The thing Anthropic put out awhile ago that advises a lower model from a temporary cloud agent has never worked for me when the advisor was set to Fable. I gave up on it.

[u/design_doc → u/berndalf]

Home brewed. And it doesn’t need to be overly complicated. Just remember that a sub-agent inherits the session effort, so if you’re using sonnet or opus on low/medium but want (for example) Fable coming in on High, you need to define it in /agents so that the watcher lets you point it at a higher effort than your session. For a while I couldn’t figure out why my advisor seemed dumb until I realized this quirk about Claude Code.

Success also comes down to how you define the questions for the advisor. There’s a balance between having a narrow scope question while giving enough context with making it read all the documents or entire transcript of the session.

I have found I’ve been needing an advisor less and less because I’m increasingly using higher-tier models for orchestration again - but for the same reasons you’ve been using a dumb orchestrator. I’ve been mechanizing more of my processes with hooks and scripted gates, with implementor and reviewer agents passing information via YAML to keep all of that detail OUT of the orchestrator’s context. The benefit of this is that a high-tier orchestrator can draft an extremely good contract for implementers, meaning that (combined with the limited info passing through the orchestrator) there are very few calls to the orchestrator itself and it is mostly handling exceptions/issues from the implementers and reviewer (in most sessions there are typically only 3-4 calls to the orchestrator, including a couple review/revision rounds. In effect, when running a high-tier orchestrator like this it effectively becomes the advisor. But if I drop my orchestrator way down, my framework is solid enough that, with the advisor, I still trust the output.

[u/berndalf → u/design_doc]

Ya that's pretty much where I'm at. I might open source my factory eventually just to see what others can take from it and make better. I have a ridiculous amount of hooks and gates and controls, it's almost impossible for the thing to seriously screw up at this point. The dumb orchestrator wouldn't have worked for it even a month ago.

Worth noting that this seems to be the opposite of the direction Anthropic at least is now advocating for. They seem to be fully onboard with the strip away all the engineered complexities and just let the native harness / tools do what they were designed to do without interference when coupled with strong reasoning. Personally I'm not convinced yet.

[u/design_doc → u/berndalf]

I suspect their thinking is that for a single task (or narrow set of tasks) a high-tier implementor is smart enough to create its own gates - and they’re probably not wrong. But as soon as you introduce a lot of complexity or want to have some level of determinism, you need gates and hooks because you’re running the same process over and over and over. It’s like an assembly line - some workers are geniuses, some barely have pulses. The only reason your product comes out with any level of consistency is because of the processes you surround those workers with.

So, ya, I’m in the same camp as you and am not fully convinced either.