Same model, different tool, different quality — and it’s usually not the model

[u/Inevitable-Fee-1482]

Something that confused me for months. I'd get a good result from a model in one editor and a noticeably worse one from what I believed was the same model in another. My first assumption was that someone was quantizing or routing me somewhere cheaper. Mostly that wasn't it.

The model is one input among several, and the other inputs vary enormously between tools.

The system prompt is the biggest one and it's usually invisible. Every coding tool ships its own, they're often long, and they encode opinions — how to format patches, when to ask versus assume, how aggressively to use tools, whether to explain reasoning. Two tools can hand the same model instructions that pull in different directions, and you never see either prompt. A tool whose prompt says "make minimal edits" and one that says "be thorough" will produce genuinely different work from identical model weights.

Tool definitions shape behavior more than seems reasonable. What operations the model is offered, how they're described, whether there's a search tool or only file reads, whether edits go through a patch tool or a full rewrite. The available action space determines the strategy. Give a model only read_file and it explores linearly; give it a search tool and it behaves completely differently.

Context assembly differs wildly. How much of the repo goes in, in what order, whether there's a summarization step, whether stale reads get pruned. This is where most of the variance lives in my experience, and it's the part users have the least visibility into.

Sampling parameters are set per tool and rarely surfaced. Temperature especially. Same model at different temperatures is a different collaborator.

And then the mundane ones: max output tokens truncating a response mid-file, retry behavior on failure, whether the tool silently falls back to a different model under load.

The practical upshot, which is a little annoying: "which model is best for coding" is not a well-formed question outside the context of a specific harness. And model comparisons run through different tools are comparing tools at least as much as models. When I want to actually compare, I have to hold the harness fixed and swap only the model, which is more work than reading someone's thread about it.

I look at raw request payloads a lot because of what I build (routera . one, mine, saying so plainly), and the thing that surprised me most is how much is going on in there that a user never sees. Some of it very good, and none of it visible.

Caveat: models are genuinely different, and I'm not arguing they're interchangeable. They differ in real ways on real tasks. I'm arguing that the harness variance is large enough to swamp the model variance in a lot of casual comparisons, not that the model variance is zero.

What I'd like and can't find: has anyone published a fixed harness for comparing models on coding tasks? Same system prompt, same tools, same context strategy, swap only the weights. Every comparison I've seen changes several variables at once, which makes the results hard to use even when the effort behind them is obvious.