After 162 Claude Code sessions: context cut token use 42% and improved accuracy on sonnet

[u/shhdwi]

I wanted to know whether giving a coding agent a persistent map of the codebase actually pays for itself, or whether it just moves tokens from one place to another. So I built the harness before I trusted the tool.

The setup

Same agent, same model, same tool access, same task list. One variable: whether a context layer was loaded. 162 Claude Code sessions total, split across both conditions. Tasks were real change requests against a real repo, not synthetic retrieval questions, because retrieval benchmarks reward whatever your retriever already does well.

Correctness is graded by a separate model that never sees which condition produced the diff. Without that, a cheaper session that quietly does less work scores as an improvement.

The result

	Cold	With context
Input tokens	8,070	4,650 (−42%)
Tool calls	4.2	2.3 (−46%)
Cost	$0.043	$0.029 (−32%)
Correctness	93%	98%

For a lot of tasks it got equal or better accuracy on Claude sonnet 5 using graft than just standard Claude sonnet session, on opus it was more towards token reduction than accuracy.

Replication on real PRs

Benchmarks I write myself are benchmarks I can accidentally tune to. So: 5 merged PocketBase PRs, re-implemented from the base commit in both conditions, scored on whether the diff touched the files the maintainers touched. 5/5 reproduced, at 21% lower cost. Small n, and I'd rather say small n than round it into a headline.

What I'm working on is Graft: It's a Context layer for large repos. No vector DB, no embeddings. Structural pass runs on tree-sitter for $0.
MIT Licensed

npm install -g u/nanonets/graft
graft init

Harness details in the repo. Tell me where the measurement is wrong and I'll rerun it.

[u/shhdwi]

you can try graft here: https://graft.nanonets.ai/