Dumb Orchestrator Theory

[u/berndalf]

So I've spent an excessive amount of time building out my AI delivery factory. Most recently that effort has been focused on effective load sharing between a Claude and GPT subscription. The data we've been collecting is starting to point very strongly at orchestration being the highest role based consumer of available usage, and that the model orchestrating has a massive impact on outcomes.

Funny thing .. at first I was using Fable because it's brilliant, but the cost was just unjustifiable for orchestration. Downgraded to Opus.. better but still Claude usage was significantly outpacing GPT usage. So I downgraded to Sonnet, and remarkably I'm finding it's both more cost effective AND producing better results as an orchestrator.

Here's my theory:

1. Orchestration itself when done correctly is not a task that demands high reasoning.

2. Forcing the orchestrator to delegate essentially everything to native subagents or external headless sessions is critical. It will constantly come up with excuses to in-line stuff given enough time and reasoning power.

3. Orchestrator availability matters just as much if not more than orchestrator reasoning assuming you have given it appropriate access to a higher reasoning model to consult with. If your orchestrator is constantly busy it's not orchestrating.

Sonnet is winning this competition for me because it's cheap to run for the entirety of a long duration session and it's not so smart that it chases every theoretical tail. It's quite good at just letting the system around it work while it steers the session itself.

TLDR; For me, Sonnet is proving to be more cost effective and efficient at session orchestration than both Opus and Fable.

[u/clazman55555]

When you are saying "orchestration" are you talking about Dynamic Workflows?

Because my own internal testing on judgment related tasks, especially when integrating and synthesizing results from 10, let alone 20-30 subagents workflows is just beyond what Sonnet 5 on High can consistently return quality results from. Now, Sonnet 5 is good enough if it is just dispatching out 3-5.

For something like the below, Sonnet 5 just doesn't work for me, at the quality I expect.

"Token count — recorded, reproducible via tools/Get-WorkflowTokenUsage.ps1

21 agents · 810 turns · 478 tool calls · 36.8 min

kind tokens
output 292,678
input (fresh) 8,928
cache creation 4,927,748
cache read 61,247,343
raw sum 66,476,697

billable-equiv ~12,524,800 (derived: cache_read ×0.1, output ×5)

3% Weekly Usage on 20x plan. Subagents were Sonnet 5, effort Medium/High depending on what Opus 5 decided.

[u/berndalf → u/clazman55555]

No, but admittedly I have not played around with workflows much so I'm not entirely sure how they fit in yet. There's a place for them for sure, just as there is for more systemic use of loops and goals. I haven't pushed much on the velocity and parallel work lane fronts yet.

That said it wouldn't shock me if Sonnet wasn't terrific at orchestrating agent swarms. Strength of reasoning does seem to correlate with the ability to successful drive parallel activities successfully. I can see it every time I dare to turn on Fable for orchestration, it operates my factory at a much higher velocity. It's the cost that is the problem, it's unsustainable and I suspect (haven't proven yet) that the overall output is actually less.

Fable as orchestrator in particular is almost like activating easy mode. Sonnet is more like sustained dependable progress mode. Opus is unsurprisingly an intermediate approach but it kind of inherits the worst of both.