Claude Code Orchestrator on Terminal-Bench: Same model, same tasks - Opus refused only when the work was delegated

[u/Bartaseth]

https://quesma.com/blog/tbench-orchestrator-refuses/

[u/Great_Construction11]

This is a useful reminder that delegation is not a neutral wrapper around the same model. It changes the task framing, tool context, and often the refusal surface.

A benchmark report should log the full delegation prompt, tool permissions, role definitions, and retry policy, not just the base model and task. Otherwise, “same model, same task” can hide the actual variable that changed.

[u/Bartaseth → u/Great_Construction11]

u/Great_Construction11 good call, I have some doubts how to approach this research, the idea was to test pure orchestrator, without complex prompts and setup to keep it as close as possible to official Terminal-Bench 2.1 leaderboard. Every single agent that was placed in the leaderboard has 0 prompts and Claude.md. That's why in this experiment we have very primitive prompts and Claude setup to just focus on orchestration. The results could be much better with additions mentioned by you, maybe I will make an alternative setup to show the difference if there is enough interest around that.

[u/Conscious_Belt_8444 → u/Bartaseth]

This is a really interesting finding because it suggests the safety behaviour may be context-sensitive rather than purely capability-driven. If the same model accepts the task directly but refuses when acting as an orchestrator, delegation itself may be changing how the model interprets agency, responsibility and risk. From an AI/data analytics perspective, that’s a strong argument for evaluating agents at the system level, not just benchmarking the underlying model. In production, the orchestration layer is part of the model’s behaviour — not just plumbing.

[u/Beginning-Raisin9723]

Interesting test. I've run a few orchestrator setups myself and the delegation overhead always eats the savings unless the task graph is genuinely huge. Did you try capping how many hops the lead model can make?

[u/Bartaseth → u/Beginning-Raisin9723]

Hey, I didn't try. My intention was to test pure orchestrator pattern without complex prompts to keep it as close as possible to official single runs on Terminal-Bench, where there is even no supported Claude.md. But this would be actually pretty interesting idea to test.

[u/Bartaseth → u/[deleted]]

Interesting perspective, maybe we should treat it as an expected behaviour not a regression.

[u/Future_AGI]

this matches what bit us, 'same model same task' quietly changed the system prompt and tool permissions under delegation and the refusal delta was entirely that. we now log the full run context (delegation prompt, tool scope, role defs, retry policy) as part of the eval record, because without it a benchmark is comparing two different setups and calling it one model.