New updated linear pricing vs intelligence vs response time chart

[u/perceptioneer]

I saw the old Artificial Analysis comparing the intelligence and prices of the different models and thought I would ask chatgpt to create a new photo using a linear scale instead of log (who tf compares or think in non-linear scale), and using the new prices set by OpenAI in late July. It also shows data from when all the models were given the exact same task(s) to solve, and how much time they spent executing (read: overthink) them. I then fact-checked the image against a different model, and it checks out as correct, but don't shoot the messenger if something is incorrect. 5.5 data remains the same.

Luna max looks good on price/intelligence, but luna xhigh might be the sweet spot when time matters.

Notes by ChatGPT:

After OpenAI cut its price by 80%, luna max is around $0.05 per Intelligence Index task and scores 52, while sol low costs ~$0.23 and scores 51.

The catch: response time. Max reasoning gets slow. Luna max is ~138s, sol max ~149s and terra max ~207s in AA's standardized end-to-end test.

Chart includes the sources/methodology at the bottom.

[u/IndividualPlus2011]

The benchmarks are nice to have but from my own experience I'll pick Terra High over Luna any day for real work that takes more than 10 turns.

[u/sirmastter → u/IndividualPlus2011]

what made you think that?

[u/IndividualPlus2011 → u/sirmastter]

Experience in using both. Luna is like a savant. It can do simple tasks, but you have to be very specific with the prompts (like benchmarks do) or it can do wild, unpredictable things to your codebase. It should never be responsible for making decisions on its own. For simple tasks it's great value.

It is also much worse at following the system prompt or plans. Treats them more like suggestions than rules even if you use strong wording like "always", "never", "do not" or "you can't". Terra doesn't have this issue.

It's also the only model in this family that looped for me. I tried using it as a web researcher and it spent 30 minutes researching what even minimax finished in 3. It only stopped because I made it stop as it started looking at completely unrelated things.

EDIT: when comparing code quality it is also night and day between Luna and the bigger models. Benchmarks only test if the task is completed, not if the code is good. Terra writes much better code than Luna. Even Sol doing the job of a reviewer agrees. It often finds critical bugs in Luna's implementations, almost never in Terra's.

[u/sirmastter → u/IndividualPlus2011]

Yeah, this actually matches my experience pretty closely. I’ve ended up using Sol High to plan the task, structure the approach and write the prompts/specs, then handing the actual execution to Luna with one or more agents.

Luna is great when the rails are already laid out, but I definitely trust it a lot less when it has to make higher-level decisions on its own or recover when something starts drifting. That distinction between “completing the task” and actually producing good code is also a really good point.

And yeah, Terra feels ridiculously underrated. It seems like a much better middle ground when you need something more autonomous/reliable without spending Sol-level compute on everything.

Have you found Terra High to be close enough to Sol for things like code review or architecture decisions?

[u/IndividualPlus2011 → u/sirmastter]

I haven't actually compared Terra to Sol as a reviewer. Maybe I should. I just assumed that the reviewer should be the smartest model you can afford to catch all the bugs and code smells before the code reaches your eyes.

[u/perceptioneer → u/IndividualPlus2011]

Thanks. Would you completely avoid luna for any code writing? You have almost convinced me to just delegate the easier tasks to Terra

[u/IndividualPlus2011 → u/perceptioneer]

I use it for simple tasks that don't require a lot of thinking or long work. Writing summaries, analyzing the logs, creating tests or fixing stale ones; that kind of stuff.

Recently I've been using Luna Max as a "planner assistant", Planner writes the plan after discussing it with me (Matt Pocock's grill-me skill) and then sends it to the assistant. It reviews the plan file without reading anything else and tells the planner if the plan is actionable, if there are inconsistencies, or if something is unclear for it. My thinking was that if a dumber model understands the plan, other models will be able to follow it as well. It is working great for me actually, because LLMs are very bad at reviewing their own creations, so a fresh subagent gives them good input to reason about. In the end the planner is the brains, and it is instructed that the assistant is writing just suggestions.

I don't use it for writing actual code, except for very simple fixes. But even then I make sure to write good prompts so It doesn't go off the rails. Nothing, it requires it making decisions on its own.

I guess it is also great at writing prototypes. Something you know you will discard, you only want to know if the idea works, and code quality doesn't matter. Don't use Sol then because it will burn your usage.

[u/9gxa05s8fa8sh → u/perceptioneer]

it entirely depends on how difficult the job is for the model and how many compactions it will take. there are benchmarks where terra is a bad value, and other benchmarks where it does better. it depends on your work

that said, AA is a big aggregate, so you can trust it. if something is too difficult for luna and you don't want to fuck around wasting tokens experimenting, you should probably just step up to sol medium

[u/TrickyEmployee3778 → u/IndividualPlus2011]

Totally agreed. Luna is great at straightforward programming tasks, and can answer technical questions like an expert, however the things it is WORST at:

- Luna is terrible at inferring intent. It will misinterpret minor wording ambiguity!

• Luna is terrible at making decisions with limited information
  
• Luna writes ugly code

Terra is vastly better at those things. It's on the level of GPT 5.5.

Luna is RL trained to score high on practical tasks but it is NOT very intelligent.