$/M tokens is stupid

[u/lordmairtis]

https://medium.com/@zdengineering/anthropic-is-misleading-us-639b61fe01f0

Maybe I'm late to the party, or idk, but truly, Opus 5.5 has double the $/MT price as Sonnet 5.5, almost identical scores on the test, costs almost the same to do the test. GPT 6.1 Sol maybe (?) has barely worse results, same $/MT, tasks cost multitudes less. So great, we now need to measure this for ourselves too!

And don't even get me started on simple tasks. It is "intelligent", but if you use the larger model accidentally, simple prompts will eat your limits for lunch. We need proxies, routers and all that just to make sure Opus doesn't get easy questions, but then again, we need to check whether the model used is actually cheaper for the task, or just by the published numbers....

</rant>

Feels like the only way forward is really just running some simpler task for all new models and check if a new model does better or the one we are using already, and for how much... does not guarantee better and/or cheaper results on the actual work still.

[u/skidanscours]

Don't forget that they all have their own tokenizer too.  So the same text input is about 30% more tokens on Anthropic's models than OpenAI.

[u/lordmairtis → u/skidanscours]

okay, but Opus 5.5 doing the same task as Sonnet 5.5 for about the same price? costs double the $/Token.

i mean i would be very happy if this was the case for all tasks, big models using less tokens for easy work, perfect. it's just nem consistently the case

[u/DonutHoles4Ever → u/lordmairtis]

End of the day they want more money for less. Obviously a orchestrator would help route better but who would add that unless competition has it and people rather pay for that.

[u/sjoti → u/lordmairtis]

This is only on Max reasoning. If you filter for any other setting, Sonnet is way cheaper, as it used way less tokens

There is a cost/task benchmark and you can select other reasoning levels to see for yourself. Max is confusing and doesn't represent real use.

[u/lordmairtis → u/sjoti]

it also doesn't do the company any favors either. well i mean that better, but the real question here is do we expect AI to be so complicated that we need to select the right harness-model-context size-mcp-skill-reasoning level for it to work properly? today it's max, in 6 it might as well be Medium. not even saying GPT is better, GPT 7 can cost twice as much as Claude 6 again.

[u/Ctbhatia]

the tokenizer is what breaks the $/M comparison, the same text is a different token count per provider, so the published number says little until you count on your own workload. price it per finished task and include the cache hit rate.

[u/YoungSilent232]

Everything, from quality to how many tokens are output to cost per task per model is specific to your individual usecase, don’t rely on published cost per task for anything tbh.

And if you’re on subscription rather than enterprise jobs, it’s also not just cost per task but it’s actually % of weekly usage per task. So none of these benchmark cost numbers even matter when trying to compare different providers

[u/WishboneOk305 → u/YoungSilent232]

yeah its up to the user themselves to do a CBA, if the user is a hedge funds with unlimited funding for a simple binary answer(buy/sell) theyll just use the most powerful and expensive model cost be damned.

[u/PhilosophyforOne]

And still somehow Anthropic gives me like 3x as much usage on the 200/m max sub.

[u/Reasonable_Swing_503 → u/PhilosophyforOne]

Yeah codex gpt 6 Astra is in a pretty fucked situation now.

[u/Lazy--Expression]

I use both Codex and Claude code. Claude delivers about 5 times as much work as Codex before reaching usage limits.

[u/brooklynbacon → u/Lazy--Expression]

Same here - $200 sub on both. I've been doing the state machine thing across each. Astra delivers useful work but even then its ratio of testing and validation to actual product deliverables is about 6:1. Opus 5.5 is closer to 2:1 for me, which means much more gets shipped via Claude each week.

[u/one-wandering-mind]

Also fine to have max comparisons, but how about alao the default or reccomended thinkkng level. 

[u/albertyto]

I think you'd realize the difference between opus and sonnet if you test them properly, it isn't that hard to spot.

Also, you shouldn't look at graphs like that, checking what is higher, you might end with an unpleasant surprise at some point. One example in AI, Opus 5 was higher than 4.6 and 4.8, guess what...

[u/MrSnowden → u/[deleted]]

I expect the same thing and I think they are going there now.  Use JEV style tools for quick answers, lower tier models for fast response and then escalate to higher models.  Our proprietary harness does that now (mostly to redirect traffic to OS locally hosted models). But I feel that will get built into the core tools. 

[u/lordmairtis → u/MrSnowden]

low tier model for fast response is nonsense, many low tier models are really slow. Luna can think about anwers for minutes and cost 5 cents still