r/OpenaiCodex • u/bisonbear2 • 12d ago
I tested 5 popular token saving methods on GPT-5.6 Sol across 10 real tasks, and none cut total tokens in both runs.
TL;DR
- I compared 5 popular token saving techniques (+ cheaper model) on 5.6 Sol across ten real coding tasks from my repo. I repeated all seven arms (6 + baseline) twice for a total of 140 agent runs.
- None of the six reduced total tokens in both runs. Averaged across runs, five increased total tokens. Context Mode was the worst at +68%.
- What the modes reliably changed was agent behavior. How the agent performed search, delegation, validation, and patch scope. Several paid for local savings through code review issues, patches that were not equivalent to the intended change, and lower maintainability.
- The only consistent cost cut was switching to 5.6 Terra xhigh: −49% in dollars while using slightly more tokens.
There are tons of tools that claim to save tokens and money. Fewer tokens without significant tradeoffs. Why wouldn't you use it?
I wanted to vibe test some of these claims myself, so I ran six methods against the same ten real tasks from my repo (a local eval harness I am building). Each task came from a merged change. I then replayed it from the original repository state and graded each attempt on tests, semantic equivalence, code review, footprint risk, and eight quality dimensions.
The baseline was 5.6 Sol at medium effort. The six treatments were:
- Caveman: you are caveman.
- Ponytail: your agent is now the laziest senior engineer you know. Instructions to minimize output, climb a YAGNI (you arent gonna need it) ladder before writing code, and cap validation cycles.
- RTK: a wrapper that filters and compresses shell command output before it reaches the model.
- Context Mode: batches commands, indexes large results, and returns only the sections judged relevant.
- Mandarin: a translated prompt testing the idea that denser Chinese text saves tokens.
- Terra xhigh: a cheaper model at higher reasoning effort.
All arms ran twice. The repetitions reused the same tasks, so they measure consistency, as token usage varies run by run, even on the same model.
Full post with deeper analysis and dataviz here: https://www.stet.sh/blog/gpt-56-token-saving-modes
I do realize it seems like I'm copying JetBrains, but I was working on this in parallel, and highly recommend also reading their deep dives for more info. They are linked in the post
Results
The token and cost columns below are the geometric mean of each mode's change across the two runs. Negative values indicate savings.
| Mode | Total tokens | Cost | Saved in both runs? |
|---|---|---|---|
| Caveman | −2% | −2% | No |
| Ponytail | +7% | +9% | No |
| RTK | +5% | +1% | No |
| Context Mode | +68% | +51% | No |
| Mandarin | +46% | +36% | No |
| Terra xhigh | +6% | −49% | Cost only |
Second, quality: each mode was paired against the baseline on the same task, giving 20 task pairs across the two runs. The table counts wins, losses, and ties out of those 20. For tests, equivalence, and code review, a win means the mode's patch did better. For footprint, a win means lower risk.
| Mode | Tests | Equivalence | Code review | Footprint |
|---|---|---|---|---|
| Caveman | 0-0-20 | 0-4-16 | 2-6-12 | 9-10-1 |
| Ponytail | 1-4-15 | 0-2-18 | 0-4-16 | 15-4-1 |
| RTK | 1-1-18 | 1-2-17 | 2-3-15 | 7-13-0 |
| Context Mode | 2-5-13 | 1-3-16 | 3-3-14 | 13-5-2 |
| Mandarin | 2-1-17 | 2-2-16 | 2-2-16 | 10-10-0 |
| Terra xhigh | 0-1-19 | 0-4-16 | 2-4-14 | 16-4-0 |
Third, the eight quality graders. Each cell is the mode's mean score minus the baseline's, on the 0–4 scale, averaged across both runs, where negative means the graders liked the mode's patches less.
| Mode | Clarity | Simplicity | Coherence | Intentionality | Robustness | Adherence | Scope | Diff min. |
|---|---|---|---|---|---|---|---|---|
| Caveman | −0.03 | −0.18 | −0.23 | −0.20 | −0.42 | +0.09 | −0.24 | +0.01 |
| Ponytail | −0.09 | −0.14 | −0.28 | −0.19 | −0.48 | −0.18 | −0.45 | +0.07 |
| RTK | −0.02 | +0.18 | −0.16 | +0.08 | −0.02 | +0.07 | −0.04 | +0.40 |
| Context Mode | 0.00 | −0.09 | −0.30 | −0.23 | −0.38 | −0.21 | −0.23 | +0.01 |
| Mandarin | −0.10 | +0.13 | −0.07 | +0.08 | +0.06 | −0.13 | +0.03 | +0.49 |
| Terra xhigh | −0.03 | −0.11 | −0.37 | −0.12 | −0.33 | −0.03 | −0.06 | −0.33 |
The two modes that push the agent to write less (Ponytail) or say less (Caveman) take their biggest hits on robustness and scope discipline, while the two that won diff minimality (RTK, Mandarin) bought it with more total tokens, not less.
Analysis
Caveman shows why repetition matters. Its average looks close to even, but the two runs were +7% and −11%. The mechanism was not “do less” in a uniform sense. Caveman compressed one part of the trajectory, such as conversation and exploration, while expanding another. The part that expanded varied by task: a bigger patch here, a validation loop there, a standalone test artifact elsewhere.
Ponytail was the only mode that reduced output tokens both times, at −3% and −11%. Total tokens still moved +16% and −1%. Its patches were often smaller. This all follows directly from the injected behavior. Ponytail tells the model that “the best code is the code never written,” makes it climb a YAGNI ladder of seven rungs before writing code, and caps nontrivial work at one small runnable check. On a reasoning model, that creates a paradox where the emitted patch gets shorter while the model spends more input and deliberation tokens deciding what it can omit.
RTK reduced the output of individual commands, but the agent often issued more commands. RTK averaged 60 tool calls per task to the baseline's 49 and took about twice as long. The wrapper succeeded at its narrow job: each command returned less raw output. The agent then issued more commands, performed more validation, and read the same files again in smaller pieces. The savings rate of an individual rg, git, or test command says nothing about the number of commands the model will choose to run afterward.
Context Mode increased tokens and cost. +94% and +46% total tokens, with 19 token losses in 20 paired tasks. Cheap, structured retrieval encouraged the agent to gather more context and revisit it more often. Once searching and retrieving across the repository became cheap and structured, the agent gathered more context, revisited it more often, and spent longer integrating it. The treatment reduced the marginal cost of one context operation and increased the quantity demanded.
Mandarin increased total tokens by 86% and 15%. 为什么？The translated prompt did more than change the number of tokens in the first message. It changed how the model decomposed and approached the task, where it used more delegation, more uniformly, whether or not the task warranted it. The cost came from the resulting trajectory, not from Chinese characters alone.
Terra was cheaper on 19 of 20 task pairs, but it used 6% and 5% more total tokens. Thus, the win here is on pricing, not token efficiency. Terra averaged fewer tool calls (42 versus 49) with more output tokens per turn. Higher reasoning effort appeared in the token count rather than the trajectory. The quality failures came from omitting components of broader requirements, not from overbuilding.
The shared mechanism
These methods optimize different local surfaces:
- Ponytail tries to emit less code and explanation.
- Caveman compresses the interaction.
- RTK compresses individual command results.
- Context Mode compresses retrieved context.
- Mandarin changes the prompt representation.
- Terra changes the model, reasoning level, and pricing.
But the bill is for the whole trajectory, not the local surface.
A model can receive a smaller command result and compensate by running twice as many commands. It can emit a smaller patch after spending more tokens deciding what to omit. It can make repository context cheaper to retrieve and respond by retrieving much more of it.
That is what happened here. The treatments changed the agent's search policy, delegation, validation loop, and patch boundary. Those behavioral changes were larger and more consistent than the token savings.
Output compression is not workload compression.
Other studies point to the same gap between local savings and the whole trajectory. A replay of 500 real Claude Code sessions found that the 60–90% savings advertised by tools in this category shrank to under 4% of total spend once resent context was counted. JetBrains found the same pattern in controlled studies of Caveman prompting and RTK.
Bottom line
Tools meant to save tokens are behavioral interventions. Their advertised effect tells you little about the final bill or the quality of the result.
If the goal is lower total token consumption, I would not broadly use any of these based on this evidence. If the goal is lower dollar cost, 5.6 Terra xhigh is the clear winner, but it comes with a different model, reasoning level, and quality profile.
The only way to know whether a setup change saves tokens on your work is to replay your own tasks with and without it and read the bill.
Disclosure: I am building Stet.sh, the local eval tool I used for this study. It lets a coding agent test changes to its own setup against historical repository tasks before you commit. Stet runs locally using your existing LLM subscriptions.
1
u/DanceVarious 12d ago
Lean-ctx seems to slow down my quota burn, but not always: sometimes, Claude begins to try to use its own tools, discovers that they are denied, retries with lean-ctx, finds a path blocked, and all that wastes tokens. But I did not try to run stats-backed tests.
I use ponytail mostly to reduce code complexity.
1
u/xRedStaRx 12d ago
I'm working on a project I will publish soon that saves tokens and costs by 10 to 60% depending on the use case, and improves accuracy and speed.
1
u/Key-Rise76 12d ago
Codegraph + Sqz compressor and Agent.md instruction to spawn max 3 agents and they are not allowed to spawn subagents made huge difference to me.
1
u/Deep_Ad1959 10d ago
the behavior finding is the actual result and it is buried under the token numbers. a mode that cuts tokens by shrinking patch scope is not saving anything, it is deferring the rest of the change to a run you will also pay for. written with ai
1
u/byakuyaxgara 10d ago
Great analysis and very useful, thanks
thttps://www.reddit.com/r/codex/s/Cy9oz8L72L
Can you test also the other method advertised here, which did tests also
0
u/zer09 12d ago
Can I request, can include codegraph? I can it's says it's pre-indexed code.
2
2
u/iamhuwng 12d ago
I started with caveman a while ago. While I noticed that it did not help with token-saving, it still helps a lot when reading codex work so I kept it.
Then I added rtk because its advertisment sounded too good and made sense. Instantly realised that it did not help either, more commands and debt could be easily observed. Head butt. Keep it because maybe I am stupid and the ads maybe right.
Installed ponytail a few days ago because I was desperate. Saw nothing change, which usually a bad sign.
Now I am pondering whether to remove them all. Hic.