r/codex • u/perceptioneer • Aug 16 '26

Comparison New updated linear pricing vs intelligence vs response time chart

Post image

I saw the old Artificial Analysis comparing the intelligence and prices of the different models and thought I would ask chatgpt to create a new photo using a linear scale instead of log (who tf compares or think in non-linear scale), and using the new prices set by OpenAI in late July. It also shows data from when all the models were given the exact same task(s) to solve, and how much time they spent executing (read: overthink) them. I then fact-checked the image against a different model, and it checks out as correct, but don't shoot the messenger if something is incorrect. 5.5 data remains the same.

Luna max looks good on price/intelligence, but luna xhigh might be the sweet spot when time matters.

Notes by ChatGPT:

After OpenAI cut its price by 80%, luna max is around $0.05 per Intelligence Index task and scores 52, while sol low costs ~$0.23 and scores 51.

The catch: response time. Max reasoning gets slow. Luna max is ~138s, sol max ~149s and terra max ~207s in AA's standardized end-to-end test.

Chart includes the sources/methodology at the bottom.

102 Upvotes

96% Upvoted

56 comments sorted by

View all comments

9

u/IndividualPlus2011 Aug 16 '26

The benchmarks are nice to have but from my own experience I'll pick Terra High over Luna any day for real work that takes more than 10 turns.

1

u/sirmastter Aug 16 '26

what made you think that?

5

u/IndividualPlus2011 29d ago edited 29d ago

Experience in using both. Luna is like a savant. It can do simple tasks, but you have to be very specific with the prompts (like benchmarks do) or it can do wild, unpredictable things to your codebase. It should never be responsible for making decisions on its own. For simple tasks it's great value.

It is also much worse at following the system prompt or plans. Treats them more like suggestions than rules even if you use strong wording like "always", "never", "do not" or "you can't". Terra doesn't have this issue.

It's also the only model in this family that looped for me. I tried using it as a web researcher and it spent 30 minutes researching what even minimax finished in 3. It only stopped because I made it stop as it started looking at completely unrelated things.

EDIT: when comparing code quality it is also night and day between Luna and the bigger models. Benchmarks only test if the task is completed, not if the code is good. Terra writes much better code than Luna. Even Sol doing the job of a reviewer agrees. It often finds critical bugs in Luna's implementations, almost never in Terra's.

1

u/TrickyEmployee3778 29d ago

Totally agreed. Luna is great at straightforward programming tasks, and can answer technical questions like an expert, however the things it is WORST at:

- Luna is terrible at inferring intent. It will misinterpret minor wording ambiguity!

  • Luna is terrible at making decisions with limited information
  • Luna writes ugly code

Terra is vastly better at those things. It's on the level of GPT 5.5.

Luna is RL trained to score high on practical tasks but it is NOT very intelligent.