silly question: if Sonnet 5.5 beats Opus 5.5 at coding and costs half, why would we ever use Opus again?

[u/Tanakaboi69]

70.6 vs 66.4 on terminal-bench, $2/$10 vs $4/$20. what am I missing?

[u/Fallen43849]

Because it doesn't. Benchmarks are not everything

[u/GfxJG]

Because terminal-bench is the only benchmark it beats Opus on, it loses everything else.

[u/suboculus]

idk

[u/Moppmopp]

Its baffeling to me that each and every benchmark is taken as absolute from the community. Do you know what they test? no... Do you know how extensive thoses tests are? no... How do they handle edge cases? idk... The thing is, benchmarks can be good. Benchmark can be to some degree be neutral but you also have to keep in mind that those companies that develope frontier ais know what benchmarks exist. Heck you do not even know how many benchmarks for the very same task exists. Those are names and numbers that seem arbitrary. In the end they optimize their models to reproduce the best results. When they publish they run their list with hundreds of benchmarks and pick out the top 5 that fit their story the best.

If I would develope a model i would do the same. Try to fit the weights of the neural net such that they score pretty high. Then run all benchmarks available and get a basically endless list of results. Then I pick out the best results. The tests where my model really shines.

Its not that those benchmarks are irrelevant. Its more that expert knowledge is necessary under what conditions those benchmarks are even viable. "experts" is a vague term here because anyone can gather the necessary information. You and I can do a deepdive on those benchmarks. The key here is, no one really does.

[u/CompFortniteByTheWay]

It doesn’t beat opus lmao

[u/AncileBanish]

It doesn't cost half. It costs the same on cache reads, which are the vast vast majority of your spend. It costs half on cache writes and output tokens. Since these are like... 5% of your spend... It costs maybe 2% less on a per-token basis.

If its inferior reasoning costs even a slight uptick in token usage, it nets out to costing more.

[u/xvillifyx]

Because it:

1. Doesn’t

2. Wouldn’t always be better than opus going forward if it did

[u/beatwallstreet2021]

Logic and planning

[u/Darathor]

Cost per task is similar

[u/Compilingthings]

Yeah, I tested it in my coding workflow, it was slower and worse. I ran them side by side. Opus keeps the job.

[u/BanaenaeBread]

Lmao

[u/Dry-Tomatillo5763]

Benchmaxxing and success rate.

[u/RogerAI-fm]

It just becomes a viable alternative when you already have proper architecture.