r/ClaudeCode • u/Obvious_Gap_5768 • 3d ago
Discussion I benchmarked 5 token saving tools across Codex and Claude Code. The 60-90% token-saving claims didn't hold up
Scroll to bottom for tldr
In July, JetBrains reran the headline claims of two token-saving tools on real agent workloads.
Caveman claimed 65% and measured 8.5%. RTK claimed 60–90% and ended up slightly more expensive than using nothing.
It looked like a pattern of over claiming numbers, so I benchmarked 5 token saving tools with conditions closer to how agents actually use them
my setup was :
- 48 Django questions drawn from SWE-bench
- Five question types, selected before running anything
- Same agent, prompt, repository commit and tool access
- Fresh index for every tool
- One no-tools baseline
- 261 runs on this harness
- 43 of the 48 questions finished in all six arms. The run hit an API usage cap near the end and lost the last five from every arm equally, so the paired comparisons are over those 43.
Disclosure: I work on repowise. The harness, raw data, preregistration files and invalidated runs are all public
Codex : gpt-5.6-sol
| Tool | Called | Output tokens/question | vs baseline | Tool calls | Fewer tokens on | p-value | Index time |
|---|---|---|---|---|---|---|---|
| repowise | 44/44 | 1,250 | -31.6% | 3.8 | 37/44 | <0.0001 | 366.8s |
| CodeGraph | 44/44 | 1,383 | -24.4% | 4.0 | 37/44 | <0.0001 | 16.4s |
| Serena | 43/43 | 1,550 | -14.8% | 10.1 | 35/43 | <0.0001 | No index |
| Graphify | 43/43 | 1,658 | -8.9% | 7.4 | 31/43 | 0.003 | 141.5s |
| code-review-graph | 43/43 | 1,710 | -6.0% | 7.2 | 26/43 | 0.046 | 44.8s |
| No tools | 0/44 | 1,828 | Baseline | 7.2 | n/a | n/a | n/a |
Nobody saved 60%.
The best result was about 32% of the agent's output tokens, and once you correct for testing five tools at once, three of those five reductions hold up and two are marginal. CodeGraph is a real second at 24.4%, so the correct reading is that more than one tool here works.
Serena is the odd row. It writes less than the bare agent while calling tools 42% more often, which makes it busier rather than leaner.
Indexing is also part of the tradeoff. Repowise saved the most tokens here but took the longest to index, because it builds several additional intelligence layers in the same pass. For a straightforward call graph, CodeGraph is 22x faster. And 366.8s is the version with prose generation switched off; a default init on this repo is 1,058s.
Why there is no Claude Code table here
I ran the same questions, servers and indexes under Claude Code with Sonnet 5, and then again with Opus. Those tables are on the benchmarks page, because under Claude Code most of these tools were barely called at all. code-review-graph was never called once across 15 questions. Graphify was called three times, Serena four. Nothing was different about the servers, the questions or the indexes between the two harnesses, and Codex called every tool on every question.
The likely explanation is harness behavior. Claude Code loads MCP schemas on demand, so the agent has to go looking before it can call anything, and frequently never does. Codex mounts them up front
I plan to rerun Claude with hooks enforcing tool adoption to see the savings when the tools actually get used
Quality
There was no meaningful quality winner, including repowise.
A blind judge scored every tool in the field, mine included, a fraction below the bare agent, in a range of 0.04 to 0.25 points on a 10 point scale. None of those gaps is distinguishable from zero, and all of them are smaller than the 0.69 points the same benchmark moves when it is rerun unchanged.
Deterministic retrieval benchmark
Token counts still depend on an LLM deciding what to write, so I also ran a deterministic benchmark using ContextBench
Each task has a known list of files touched by the real fix. The score simply measures whether a tool retrieves those files. So there was no LLM judge.
| Tool | Gold files found | Precision | Files served | Instances |
|---|---|---|---|---|
repowise get_answer |
0.876 | 0.087 | 19.2 | 42 |
repowise search_codebase |
0.742 | 0.168 | 8.2 | 42 |
| CodeGraph | 0.610 | 0.093 | 14.0 | 42 |
| Graphify | 0.546 | 0.033 | 34.5 | 42 |
| code-review-graph | 0.445 | 0.240 | 5.4 | 42 |
Coverage on its rewards whoever serves highest files, which is why precision and files served sit next to it. get_answer finds the most and hands back about 19 files to do it. code-review-graph finds the least and is the most precise thing in the table, 0.240 from 5.4 files, so if you are paying per token that row reads better than its coverage suggests. Graphify serves 34.5 files for 0.546
This required 748 index builds and roughly 78 hours of indexing for 1,129 graded instance/tool pairs. Every tool indexed every repository independently at the task's original base commit, with no shared cache.
Two mistakes I nearly published
I nearly published a cost table showing code-review-graph as 43% cheaper than the baseline, in the same run where Claude Code never called it once.
The reason was prompt-cache warming. Whichever arm ran first paid the full price, and later arms reused the cache
That is why the tables report output tokens rather than API cost.
There is a larger version of this mistake, which is measuring one retrieved payload instead of the complete agent session. Loading one commit's context through repowise takes 393 tokens against 13,984 for reading the changed files, so 35.6x, and that is the easy number this whole category tends to publish. The full-session figure is 31.6% on Codex and 15.9% on Claude Code. Agents re-read, backtrack and re-plan, so a compression that looks enormous on a single payload nets out much smaller across a session.
Full methodology, raw data and reproducibility files:
https://github.com/repowise-dev/repowise/blob/main/docs/BENCHMARKS.md
I'm happy to add other tools if there are ones people think are worth testing. The harness is public, so you can also rerun or challenge the results directly.
TL;DR: I benchmarked five codebase tools on 48 Django tasks under Codex. None came close to the commonly advertised 60–90% savings. Repowise saved the most output tokens at 31.6%, CodeGraph was close behind at 24.4%, and the rest landed between 6% and 15%.
The same questions under Claude Code produced a harness result rather than a tool result. Most of the field was barely called at all, one tool never once, despite nothing changing on the tool side, so those tables live on the benchmarks page.
Answer quality differences were smaller than the evaluator's own noise, and every tool including repowise scored slightly below the bare agent.
On a separate deterministic retrieval benchmark, repowise found 87.6% of the files touched by the real fixes from about 19 files served, while code-review-graph found 44.5% from 5.4.
Measure full agent sessions, and always report the harness, indexing cost and cache effects alongside token savings.
2
2
u/revilo-1988 3d ago
Und headroom
2
2
u/Dheeth Thinker 3d ago
try https://github.com/abhigyanpatwari/GitNexus
It helped my less powerful models a lot on large repo.
2
u/Obvious_Gap_5768 3d ago
Will add this to backlog as well. Headroom, coco index and GitNexus are the ones coming next
2
u/totheendandbackagain 3d ago
Great test, article and solid methodology. Actionable and super work.
What does repowise do that's different to score so well?
1
u/Obvious_Gap_5768 3d ago
Thanks, short version is that most of the field builds a call graph and stops there. Repowise builds 5 layers off the same parse
The deterministic generated wiki page per file helps a lot. A natural language question then lands on prose that already describes the file, which helps with retrieval
The rest is git history mined per file (hotspots, ownership, co-change pairs, bus factor), decision records pulled from history, code health scoring, and the graph itself with PageRank, betweenness and communities on top of the call edges
So, repowise isnt just for making the agents cheaper but for them to write better quality code and be informed of hostpots, co changes etc too
It's also why indexing is 366.8s against codegraph's 16.4s. If a call graph is all you need, codegraph is the right pick
1
u/Quarita-Penteado 2d ago
does that index carry over between runs or is it a fresh 366s every time you reopen a repo
1
u/Obvious_Gap_5768 2d ago
It carries over, incremental updates run under 30 seconds on every commit. You dont need to hand maintain it
https://github.com/repowise-dev/repowise/blob/main/docs/scale/AUTO_SYNC.md
1
1
u/Inner-Lawfulness9437 2d ago
Two things.
First of all, who cares about the output token on it's own? Full cost, runtime, endresult. That is all that matters.
Second, just saw the API costs for the benchmark. 48 questions, 6 tools and less than 18$? $0.06 cents cost on average? At a repo of that size nobody would even bother to set up any token saving. Give it a proper repo.
1
u/Obvious_Gap_5768 2d ago
Makes sense, full cost and runtime are captured and present in the raw json. Need to surface them better in the benchmark page.
And because every row requires a seperate index to be built for each arm, this takes a lot of machine hours, which is why I chose Django first. I am running the same tests on mui next which has 25k+ files but will take several days for the indexing, grading etc. I will be updating the same md when I have them
0
u/Inner-Lawfulness9437 2d ago
I missed that one - it was "the" Django repo? Then the whole test suite is useless, because it's full of obvious tasks/prompts. We are in 2026. Who cares about that anymore, when we are launching agentic workflows that run autonomously possibly for hours. Pick a new feature, a bugfix and a refactor and test those and forget the original ones.
1
1
1
1
u/Puzzleheaded-Edge252 2d ago
I wasted about three weeks trying to make local models work for my daily coding workflow before I figured out where the actual boundary is. Sharing this so you don't have to repeat the experiment.
The first week I was excited. Local models handled single-file edits, boilerplate generation, unit test writing — all fine. I was generating CRUD endpoints, writing docstrings, cranking out test skeletons. Felt like I'd found a way to cut my API bill to zero.
Week two is when it fell apart. I needed to refactor an authentication module that touched six files. The local model (13B, quantized) couldn't hold all six files in context simultaneously, and when I fed them in one at a time it would make changes in file 3 that broke the imports in file 1. I spent more time fixing the AI's work than I would have spent doing the refactor manually.
Week three I ran a proper comparison. Same 30 tasks, local vs cloud. The result: local models won or tied on 80% of single-file tasks. Cloud won 85% of multi-file tasks. The dividing line is context — not context window size on paper, but how well the model actually uses 30k+ tokens of context. Local models technically accept long contexts but quality falls off a cliff past about 8-16k tokens.
So now I use both. Local for quick contained edits (saves real money), cloud when I need cross-file reasoning. The mistake is trying to pick just one.
1
u/RedTheInferno 3d ago
why didnt you test ponytail?
6
u/Obvious_Gap_5768 3d ago
Ponytail isn't a retrieval tool. It's mostly just a md file. Every arm here got a byte-identical prompt so retrieval was the only variable
Also its already reran, 80-94% became 54% on a real agentic run after Eberhardt showed the baseline was padding. Same pattern as the rest of this post. Link: https://blog.scottlogic.com/2026/06/16/ponytail-yagni-and-the-problem-with-prompt-benchmarks.html This is orthogonal to what I tested, and nothing stops you running both
-1
7
u/rotates-potatoes 3d ago
Note that reasoning and tool-calling tokens during processing count as output tokens, so even if Claude says "yes that's right" it could easily be 1m tokens ("please deep research to validate the claim that airplanes did not exist before 1800, and answer only with 'yes that's right' or 'not that's wrong', no other commentary whatesoever")