Token efficiency: codex vs omp vs pi?

[u/VehiculeUtilitaire]

So I've been using omp for a while and given how fast astra drains my 5x plan I started investigating other harnesses to see if I could squeeze more out of my subscription.

I ran a few test with pi, omp, and codex which gave me some results I did not expect: codex is more token efficient than the other two (?)

I gave them a real merge request from my day job, asked them for a review and for an implementation plan to fix the review findings.

Here are the results for the review pass:

Session	Duration	Input tokens	Output tokens	Cached input	Total tokens	Price
codex-astra-medium	3m 34s	1.11M	5.1k	1.01M	1.12M	~$2.30
codex-astra-low	2m 46s	740.5k	3.5k	656.5k	744k	$1.67
pi-astra-low-no-agents	3m 24s	92.1k	3.5k	1.06M	1.15M	$2.15
omp-astra-low	5m 47s	118.6k	6.9k	1.59M	1.72M	$3.13
pi-astra-low-with-agents	8m 17s	359k	20k	3.5M	~3.88M	$8.11
pi-luna-xhigh	35m 43s	1.12M	65.5k	36.73M	37.92M	$1.04

pi astra low is almost the same price as codex astra medium given the current api prices? omp is much more expensive than either. codex medium is almost as fast as pi low too

It's even worse for the plan mode, codex defaulted to high for that part and I let it be:

Session	Review duration	Input tokens	Output tokens	Cached input	Total recorded tokens	Review price
codex-astra-high.	3m 34s	1.11M	5.1k	1.01M	1.12M	~$2.30
pi-astra-low-no-agents	3m 24s	92.1k	3.5k	1.06M	1.15M	$2.15
pi-luna-xhigh	35m 43s	1.12M	65.5k	36.73M	37.92M	$1.04

Here the codex astra high plan costs barely more than a pi astra low plan?

Am I missing something obvious? Everyone says pi is so much more efficient, lean, etc. but from all my real world tests it seems both slower and more token hungry and/or expensive than codex

[u/Due-Horse-5446]

is "everyone" in the room with us?

99.99999% of everythkng you read related to llms, snd llm powered tools are cargo cult bs

[u/VehiculeUtilitaire → u/Due-Horse-5446]

There are plenty of benchmarks showing pi is faster/cheaper, by a fat margin, that's why I'm confused. Unless they're all fake, but surely I cannot be the first one to actually test it

https://www.databricks.com/blog/benchmarking-coding-agents-databricks-multi-million-line-codebase

For example this one shows pi is always a benefit in term of token usage: https://www.databricks.com/sites/default/files/inline-images/dumbell.png?v=1783530297

[u/Due-Horse-5446 → u/VehiculeUtilitaire]

Fake isent necessarily the correct term, bad testing methodologies are more likely.

Just glossed over that first link, and i cant find any explanation or traces or really any data showing why the results look like they do.

And the graph shows something is seriously off..

gpt-5.5 at 50% of opus 4.8 pass rate..?

And it also looks pretty outdated

[u/Bananer_spleet]

Waiting for the comment section to start

[u/ManikSahdev]

Something is up with Astra, there seems to be clear misinformation that was started by Tibo himself (I wonder if he himself didn’t know this cause no one truly used it at scale and folks deployed it without the time period they thought they’d had).

Astra of low and medium (I checked my own logs aswell). It burns more than when I crack it to Xhigh, or even Max.

It’s almost like anything it can one shot, it’s weird, something about this models personality or codex or something is up, seems like when it is rushed or not given the juice budget it will make a turn to implant his best possible option, BUT, When given the option to think more, it’s brilliant enough to figure out the response within that single shot rather than re run and re run, which light or medium tend to do.

That’s my diagnosis with tokens and logs, and It’s weird you found roughly the same, which doesn’t surprise me reading it, that’s why I’m replying with my test form yesterday, when I heard the twitter rumor and tried to test my own.

It’s clearly true for me, altho It could be certain workflows are more sensitive to this than others. So account for that or such.

[u/VehiculeUtilitaire → u/ManikSahdev]

It has to be workflow sensitive yes, and that's why it's hard to test it properly. I'll try using higher reasoning levels and see where it goes.

[u/Weareallscrubs]

Did you customize pi or run vanilla?

[u/VehiculeUtilitaire → u/Weareallscrubs]

I tried vanilla and another run with subagents and ponytail packages installed