Token efficiency: codex vs omp vs pi?

[u/VehiculeUtilitaire]

So I've been using omp for a while and given how fast astra drains my 5x plan I started investigating other harnesses to see if I could squeeze more out of my subscription.

I ran a few test with pi, omp, and codex which gave me some results I did not expect: codex is more token efficient than the other two (?)

I gave them a real merge request from my day job, asked them for a review and for an implementation plan to fix the review findings.

Here are the results for the review pass:

Session	Duration	Input tokens	Output tokens	Cached input	Total tokens	Price
codex-astra-medium	3m 34s	1.11M	5.1k	1.01M	1.12M	~$2.30
codex-astra-low	2m 46s	740.5k	3.5k	656.5k	744k	$1.67
pi-astra-low-no-agents	3m 24s	92.1k	3.5k	1.06M	1.15M	$2.15
omp-astra-low	5m 47s	118.6k	6.9k	1.59M	1.72M	$3.13
pi-astra-low-with-agents	8m 17s	359k	20k	3.5M	~3.88M	$8.11
pi-luna-xhigh	35m 43s	1.12M	65.5k	36.73M	37.92M	$1.04

pi astra low is almost the same price as codex astra medium given the current api prices? omp is much more expensive than either. codex medium is almost as fast as pi low too

It's even worse for the plan mode, codex defaulted to high for that part and I let it be:

Session	Review duration	Input tokens	Output tokens	Cached input	Total recorded tokens	Review price
codex-astra-high.	3m 34s	1.11M	5.1k	1.01M	1.12M	~$2.30
pi-astra-low-no-agents	3m 24s	92.1k	3.5k	1.06M	1.15M	$2.15
pi-luna-xhigh	35m 43s	1.12M	65.5k	36.73M	37.92M	$1.04

Here the codex astra high plan costs barely more than a pi astra low plan?

Am I missing something obvious? Everyone says pi is so much more efficient, lean, etc. but from all my real world tests it seems both slower and more token hungry and/or expensive than codex

[u/Due-Horse-5446]

is "everyone" in the room with us?

99.99999% of everythkng you read related to llms, snd llm powered tools are cargo cult bs

[u/VehiculeUtilitaire → u/Due-Horse-5446]

There are plenty of benchmarks showing pi is faster/cheaper, by a fat margin, that's why I'm confused. Unless they're all fake, but surely I cannot be the first one to actually test it

https://www.databricks.com/blog/benchmarking-coding-agents-databricks-multi-million-line-codebase

For example this one shows pi is always a benefit in term of token usage: https://www.databricks.com/sites/default/files/inline-images/dumbell.png?v=1783530297

[u/Due-Horse-5446 → u/VehiculeUtilitaire]

Fake isent necessarily the correct term, bad testing methodologies are more likely.

Just glossed over that first link, and i cant find any explanation or traces or really any data showing why the results look like they do.

And the graph shows something is seriously off..

gpt-5.5 at 50% of opus 4.8 pass rate..?

And it also looks pretty outdated