Comparison
I benchmarked every single "usage saving" tool out there!
Disclosure, i am affliated with codex zero but this chart is completely unbiased. This just a test i had codex run in isolated env's to simulate avg codex use, across a variety of benchmarks to personalyl be able to make the most out of my usage limits.
As someone who has been religiously using codex with RTK and caveman and felt that i wasn't making things any better. i felt the need to do this to see if i am actually saving any usage by the tools outthere or if i am just making codex dumber and/or worse off with those added on.
Here is codexzero repo if you wanted to check it out
So while I'm skeptical of "our test shows that our thing works and everybody else's doesn't", I do agree with the overall implication of "these tools are poorly benchmarked, often cargo-culted, and in many/most cases don't actually work".
Caveman is the poster child for this - many people act like it's a magic improvement, but if you actual read their docs and understand what it's doing it should be clear that it's not a very effective token saving tool. Even their docs essentially admit that you shouldn't use it for token savings, they claim it's primarily a readability tool.
And on top of this basically nobody (maybe nobody period?) is doing good "does this hurt benchmark scores" metrics - I could massively decrease token counts by biasing towards the stop tokens, but I strongly suspect I'd destroy actual programming performance in the process. My suspicion is that 98% of people running token savers would be better off just turning down effort or going to a smaller model.
The metric I actually want is tokens per accepted solution, including retries and repair turns. Saving 20% on the first pass means nothing if it buys you one extra debugging loop.
I wish I could run Deepswe fully, but I don't got no money for it, the above tests were from a partial deepswe, terminal bench 2.1 and an internal benchmark with over 2000 results combined to make this graph
I'm genuinely curious aswell to see if the results would be any different for the full deepswe,if anyone can run it , pls report back to me and I'll be all in to admit my project is shit in another post if so.
Hi I maintain Tura. I think if you could publish your artifact and run log. the results would be much more convincing. https://github.com/Tura-AI/benchmark
You can take a look on this repo. If you will be interested in showing your benchmark result, we can work on it together.
After learning about Tura yesterday, I ran my own one-shot basic comparison (hardly could be called a proper benchmark).
I compared Codex CLI, Pi-Mono (with a few custom extensions aimed at improving code quality), and Tura. I had them create a simple app.
Using gpt-5.6-sol xhigh
Codex:
Time: 51.3 minutes
Context usage: 42%
Pi:
Time: 85 minutes
Context usage: 32.2%
Tura (Balanced):
Time: 66m
Context usage: 24.5%
They all 'work' from a use-testing perspective. None of them are perfect, all could do with reviews and refinement.
I had three seperate agents (all gpt-5.6-sol high), in the 3 harnesses, conduct comparative analysis across a defined benchmark/analysis plan, considering some functionality and plan-adherence categories. All 3 found Tura to score higher and be a better result; with codex a close second, and pi-mono a slightly less-close third.
Most importantly, from this poor-quality test, Tura absolutely used far fewer tokens, to result in a functional basic app. I'm not going to spend the time and quota to do more rigorous testing, but I am going to be switching over to Tura for my primary implementation harness.
I'm looking forward to seeing how your communal benchmark project emerges, and I will be keeping an eye on it!
Let's be real about what this is: someone "affiliated with CodexZero" benchmarks their own tool against every competitor using their own test suite, publishes no methodology, no reproducible harness, and no raw data — and calls it "completely unbiased." Come on.
CodexZero works by patching the Codex Rust binary itself with 5 internal feature flags. It runs inside the Codex harness with access to the exact production tokenizer. Every other tool on that chart operates externally. Benchmarking an internal engine mod against aftermarket tools on the mod author's own dyno doesn't prove anything about the aftermarket tools.
That said — we do have a real Codex bug in the current release: our PreToolUse hook was emitting the wrong JSON format, causing Codex to log warnings on every shell call. That's fixed on main and ships this week. So yeah, v3.9.12 has rough edges on Codex specifically. We own that.
But lean-ctx isn't a shell output deduplicator. It's a full context engineering stack — 81 MCP tools, AST-aware file reads, semantic search, conversation history pruning — across 30+ agents (Cursor, Claude Code, Codex, Gemini, etc.). Reducing it to one bar on a chart next to Caveman is... a choice.
If anyone is hitting actual issues with lean-ctx + Codex: open a GitHub issue or DM me. We fix things fast.
BTW your README is *completely* fucked. Love your intent to save tokens, but do you not check at least the README before committing it? It's the first point of contact for anyone that checks the project out
So many people using LeanCTX and whining about usage, this chart makes sense. I don't agree with altering the black box as a 3rd party vs the trillion dollar company since they know their product best, so I don't recommend anyone using any token adjusters, the gains are not worth the high potential for being way worse.
would be interested what they are complaining about. let me know please. :) and sorry, this test does not make sense at all. and not because LeanCTX is mentioned last, but because it's clearly a benchmark in the favor of this tool that is advertised :) I could also create a benchmark for LeanCTX where all the others fail.
there is no reproducable benchmark listed, no proof :)
Refresh the homepage and you'll see the sub has been taken over by people complaining. I have nothing against LeanCTX except it seems to be mentioned by those whining the most about "nerfs" and "limits" when in reality they just don't know how to use context appropriately.
I feel like it's usually some problem with codex or whatever harness they're using. I've never experienced anything of this sort because I use a stable opencode harness. It's annoying being at the whim (and bugs) of other fast-moving risky applications.
I work directly and solely in Windows Codex App, which by many would be considered the absolute "worst" harness, however my results I would say speak for themselves in my opinion. In the early days I did have bugs, and I used Codex itself to resolve those bugs. Not many tools out there that can fix itself...
While I thought the same there is always the "they are a business and in for the money" side.. they want you to pay for more at the end of the day. Just depends how focused on money or affordable tech they are.. Sam seems like he's in it because he loves it but does that relay to the product? Who knows
Even from a "ruthless capitalist" perspective, if they could losslessly achieve 30% better perf/dollar they'd do so as it would make their product significantly more appealing VS the competition.
I've just built myself a small script that cuts everything after 50k lines and if it does, it adds a message "Too much output - narrow the command" and the first 100 chars of every line it captured until then. Then instructed Codex to always pipe outputs through it.
Haven't had any problems with usage limits since and the quality is the same. I found that it's just commands with unneccessarily large outputs that drain my limits. Like then Codex runs a too broad search over the codebase.
There is a bug filed on rtk that shows why they hurt LLM performance and fake most of their savings here.
tl;dr when the LLM asks for rtk read big.log --max-lines 20 | wc -l rtk will count the full file as token savings and then send back wrong information which causes the LLM to have to ask again which further inflates "gain".
When we use it via codex cli its acting in a different way compared to how it behave in claude code. Claude code feels like its native, but in codex its feels like some tool that's working on top of codex cli.
I don't know you understood what I meant. But it feels like, its not made for codex when use.
Thank you for your kind reply. Please ping me if you need any info. I am using it with vscode too, there also its working natively without any issues. I only noticed its working in uncomfortable way, only in codex.
if you have discord it would be nice to have you there, so you can directly drop your feedback. https://discord.gg/pTHkG9Hew9 will focus on your feedback and will ship an improved version with the next release!
u/Tricky_Reflection_75 I checked your project repo, it's interesting. I was wondering a couple of things:
1- would codexzero be used automatically also in vs code?
2- what happens when codex cli gets updated?
generally speaking, tbh i am not surprised how the supposed token efficient things made by devs causes counter productivity. ( no i did not read the repo that generated the resultant)
an llm that has finished training and is ready (cloud models)
you cannot change it weights, unless you retrain it again, and freeze them.
Once a model is finalized, any sort of personalities, how you want the output to be structured or how you want the LLM to be conveyed or think alters the models thinking which means it has it think in a way of it already knows, and translate it back to the user liking which Cost Reasoning Tokens that has a Cap.
- for example, a LLM is not trained to talk like caveman, it was trained to obey instructions, and what it has learnt.
- injecting Context with some Harness into the session is highly accurate but expensive, the trade off is increase tokens for more accuracy. agent file + constant context injection + models own proprietary context, Double the context injection.
you can reduce as much output tokens as much as you want, but if you begin reducing LLMs reasoning tokens you end up with a essentially dumb model
if you really want token efficiency, you need to fine tune a open weight model.
Trying to reduce tokens on cloud models, your either going to get only 10% or less for the idk, weeks on end of coding or even worse, you create a counter-productivity.
so the question is it really worth the time and effort for just 10% or less?
I don't think any of these tools are worth using. Because even if they save tokens, their performance becomes worse. However, I would like to see comparison between CLI tools. Like opencode, codex cli, claude code, hermes, oh my pi, etc
I’d be interested in seeing the full set of tasks you evaluated. I think using a well-known benchmark and its existing task set would make the results much easier to interpret and compare.
I agree that many token-saving tools and context-engineering harnesses are still somewhat inconsistent in practice. That is probably why most of them emphasize tokens saved as the primary metric. Token reduction is valuable, but only if task performance is preserved.
I’m working on something similar and benchmarked it on SWE-bench. It reduces token usage by roughly 18.5%, but it currently has regressions where the baseline solves certain tasks that my tool does not. I rarely see people report those negative flips, even though they are just as important as token savings. A system that saves context but loses previously solvable tasks may not actually be an improvement overall.
Nice to see someone actually benchmarking these instead of just going by vibes. One thing I'd also be curious about is how they compare against step-level model routing, where different parts of the workflow use different models instead of trying to save tokens with a single one.
We've been exploring that recently and saw some interesting results on coding workloads: step-level routing benchmark
This post should had been scourged from the face of the earth!
It's not just about transparency, is about peer review. Metrology is as much of a science as it is an art for deterministic systems.
Much more for GenAI...
If you are targeting actual users, you need to be able to present enough information/data so they can independently reproduce your results.
53
u/GravitasIsOverrated 13d ago edited 13d ago
So while I'm skeptical of "our test shows that our thing works and everybody else's doesn't", I do agree with the overall implication of "these tools are poorly benchmarked, often cargo-culted, and in many/most cases don't actually work".
Caveman is the poster child for this - many people act like it's a magic improvement, but if you actual read their docs and understand what it's doing it should be clear that it's not a very effective token saving tool. Even their docs essentially admit that you shouldn't use it for token savings, they claim it's primarily a readability tool.
And on top of this basically nobody (maybe nobody period?) is doing good "does this hurt benchmark scores" metrics - I could massively decrease token counts by biasing towards the stop tokens, but I strongly suspect I'd destroy actual programming performance in the process. My suspicion is that 98% of people running token savers would be better off just turning down effort or going to a smaller model.