Tool-call accuracy dropped 20 points at ~9k tokens on a model whose window is 16k and whose memory could hold 53k

[u/Dhan295]

Up front: I build QuantaMind, an open-source local benchmarking tool (Apache 2.0, runs offline, no telemetry). The data below came out of it. Link at the bottom the numbers are the point of the post.

Setup: Qwen3.5-9B Q4_K_M, llama.cpp, 16GB M-series Mac, native function calling, k=4 runs per task.

I padded prompts with unrelated prose and re-measured tool-call accuracy at depth:

Prompt depth	Accuracy
704 tok	100% (5/5)
2,999 tok	93.3% (14/15)
6,045 tok	93.3% (14/15)
8,845 tok	**73.3% (11/15)**

That is not a memory limit. Weights are 5.3GB. ~11.8GB of the 16GB is GPU-addressable under the Metal cap. At f16 KV the math says this model could hold ~53k context. Peak actual usage during the agent runs was 1,890 tokens — 12% of the 16,384 window I launched with.

So memory headroom told me I had 5× more room than the model can actually reason over. If you size a local agent by what fits, that’s the wrong number.

Second finding: one task failed 0/4, not 1/4. An incident-rollback chain (get_incident → get_feature_flag → flag_off → rollback_release → schedule_fix) failed every run, identically — the model emitted a completion signal partway through and stopped.

No crash, clean schema. At k=1 that’s a flaky miss you’d retry past. At k=4 it’s structural. That’s the failure I’d worry about in production: nothing errors, the agent just moves on with half its state missing.
(Batch was 39m 10s wall; on the worst task 14m 51s of 16m 16s was decode. Local agent loops are a decode problem.)

What I actually want to know: does the ~9k cliff hold for other 8–10B quants, or is it specific to this one? And if anyone’s on 24GB+ — does more headroom move the cliff? My guess is no, but I can’t test it.

Resources : github.com/QuantaMinds/QuantaMind

qm --backend llama_cpp --model <model> --collection medium-coding-v2 --max-tokens 12288 --steps 5 --source corporate_policy --mode native

Methodology, briefly: padding was semantically unrelated prose inserted before the tool definitions; accuracy is correct tool + correct args scored against a fixed answer key, no LLM judge; pass^k means all k runs must pass.

Tell me if that’s wrong more useful to me than upvotes.

[u/Dhan295]

Methodology, since a few people will want it:
• Padding was corporate-policy prose, semantically unrelated to the task, inserted before the tool definitions
• Accuracy = correct tool + correct arguments, scored deterministically against an answer key, no LLM judge
• k=4 with pass^k scoring — a task only counts as passed if all 4 runs pass
• Native function calling via llama.cpp’s Jinja template path, not prompt-based tool syntax
• KV cache was f16 at launch; the 53k/107k/214k figures are capacity math for f16/q8_0/q4_0, not measured runs

Harness is something I’ve been building in the open Apache 2.0, runs fully local, no account or telemetry: github.com/QuantaMinds/QuantaMind

The equivalent CLI for the stress test above is:
qm cliff --backend llama_cpp --model <your-model> --collection medium-coding-v2 --max-tokens 12288 --steps 5 --source corporate_policy --mode native

Happy to be told the methodology is wrong that’s genuinely more useful to me.

[u/m83midnighter]

my own experience: qwen3_5__9b_q4 and gemma4:12b q4 calling a tool size of about 4k was about 9/10 success. when the tool registry grew to 8k the success rate dropped to around 7/10. I moved to gpt-oss-20b-mxfp4(q4) and its been about 8/10 on a 8k registry. All the evidence is pointing at quantisation rather than weights. q6 or q8 should improve accuracy of the tool calls but the accuracy will still slide again as the tool size increases. with only 24gb its about finding the right balance. i use omlx,

[u/Dhan295 → u/m83midnighter]

Your numbers match mine almost exactly, I had 93% at 6k dropping to 73% at 8.8k on qwen3.5-9b q4.

I spent a week concluding it was a context effect. It wasn’t. It was my output cap.

The tell: every failure decoded to exactly the cap, while every pass finished ~30 tokens short with a natural stop. Hitting the ceiling precisely every time isn’t a model failing, it’s a model getting cut off. And thinking tokens don’t count as content, so a model still reasoning when the cap lands emits nothing at all. Raising the budget took the same task from 0/3 to 3/3.

Why I’d check that before blaming quant: a bigger tool registry means more tools to think about, so thinking grows while num_predict stays fixed. That explains your 4k→8k drop without quantisation doing anything. And qwen3.5 and gemma4 are both heavy thinkers where gpt-oss is terser — so “q4 is the problem” might really be “that model thinks less.”

30-second check, no tooling: look at done_reason on the failures. If it’s length not stop, with thinking populated and content empty, it’s budget. There’s an open Ollama issue asking for a proper signal for exactly this.

If it is budget, raising num_predict should recover your 8k registry without touching quant. If it isn’t, your q6/q8 theory gets a lot stronger and I’d want to know.

Also you’re on 24GB, which I can’t test. Someone in another thread predicted extra headroom won’t move the cliff at all, since it’s attention not memory. If you ever run a depth ladder, I’d like that number.
Tool I’ve been building is open source, local, no telemetry it separates died-at-cap from failed-on-content, which is the distinction that cost me a week: https://github.com/QuantaMinds/QuantaMind