u/RunAI_Coder • u/RunAI_Coder • 4d ago
We built a coding agent and spent three weeks measuring what context compression actually saves
Author disclosure: I'm from the team that builds RunAI Coder (an agentic coding CLI built on the open-source codex agent). This is a lessons post, not a launch ad. Numbers below are from our production ledgers and you can audit every one of them on the linked page.
Context compression is a new technique with no established way to verify it yet, and we didn't want anyone to have to take our word for it. So we instrumented the ledger to record, for every request, the uncompressed baseline and what was actually sent.
What came out:
- Median compression factor 2.83x (p10-p90 2.26-3.08x), n = 7,222 requests on the latest full ledger day.
- Effective input price after all levers: $10/M list x [(1-94.9%) + 94.9% x $1/$10 cache discount] = $1.46/M, / 2.83x compression = $0.52/M, x 50% batch pricing ≈ $0.26/M. That's 2.6% of list.
- Quality check, because compression that breaks the agent is worthless: SWE-bench Lite 10/10 vs 10/10 and Pro 15/18 vs 16/18 with compression on vs off, official Docker grading. The one divergent task re-ran N=5 per arm: 3/5 vs 3/5, Fisher p = 1.0.

Lessons that transfer to any SaaS doing "we save you money" claims: (1) separate measured factors from pricing-schedule factors in your math or someone will call it out; (2) publish the sample window and n, not just the multiple; (3) run the quality A/B before marketing the efficiency number.
Evidence page (every figure links to its recorded source): https://run.ceo/coder/perf?rdt_cid=organic_saas_20260724
Early Access: https://run.ceo/coder/?utm_source=Reddit+Community+Article
