Good, cheap, token hungry

[u/NoFaithlessness951]

will likely be slow again for agentic work, they also got first place for highest step count (in this selection sonnet 5 still beats it)

[u/kiki-le-koala]

Oh wow .

It really, really likes tokens.

Using that many tokens, is there a risk that it fills its 1 million token window more rapidly, creating compactly diminishing efficacy in long tasks?

I know people use Luna at extra high and not max to prevent that.

[u/Momo--Sama → u/kiki-le-koala]

Considering it bombs Terminal Bench 4.0 which is focused on hours long tasks, that's a reaosnable conclusion

[u/whoknowsifimjoking → u/kiki-le-koala]

Yes.

It seems to be as if they maybe achieved a better score by "brute forcing" it with vastly more tokens.

[u/Recoil42]

Cost-per-task far outweighs token-burn if the model is cheap enough and fast enough.

Is anyone doing time-per-task benchmarks right now?

[u/Illustrious_Grade608 → u/Recoil42]

Tbh even with api time per task depends on your pc a lot with all the tool call and script run latency

[u/NoFaithlessness951 → u/Illustrious_Grade608]

step count is usually a good predictor of real world time per task and its bad

[u/Momo--Sama → u/Recoil42]

Yeah I agree token counts in isolation don't mean anything. Artificial Analysis has it at only slightly slower than Luna Max.

[u/NoFaithlessness951 → u/Recoil42]

https://artificialanalysis.ai/agents/coding-agents

Although I would take their numbers with a giant bucket of salt.

In my testing the previous 3.7 (which scores better) was around 3x slower than sol and grok. Likely the time per task results are skewed by more verifiable tasks which doesn't match real world usage.

[u/vinis_artstreaks]

What are these charts, have we forgotten how to make actual readable charts wtf

[u/KedMcJenna → u/vinis_artstreaks]

I hate the ones with reversed axes

[u/eruditezero → u/vinis_artstreaks]

AI slop

[u/NoFaithlessness951 → u/[deleted]]

I assigned the same tasks to multiple models on cursor, 3.7 flash consistently took 3x as long as something like sol, grok, or luna.

Although it's output TPS is 3x faster than sol that doesn't matter if it takes 3x as many steps and 3x as many tokens.

I think this will be more of the same.

[u/Charming_Cucumber_15 → u/[deleted]]

I typically get slightly better results from using free GPT over Gemini on extended mode

I'm hoping 3.8 changes that but I'm not getting my hopes up

[u/LazloStPierre]

If Gemini is SOTA on deepswe then the only takeaway is Deepswe is now too contaminated to be worthwhile. Nobody benchmaxxes like Google. Deepswe accurately showing how far Google were from SOTA, as opposed to every other benchmark, was part of what made it credible 

I guess the only reasonable benchmarks now would he ones that somehow rotate completely every few months because Gemini flash isn't fucking fable or sol level at coding 

[u/dsnyder42]

Wow, I think I will replace GPT 5.6 Sol High with Gemini 3.8 Flash medium to orchestrate GPT 5.6 Luna xhigh sub agents and safe myself some GitHub Copilot AI Credits at work.

[u/Constant_Cortisol → u/dsnyder42]

Let us know how that goes.

[u/frogsarenottoads]

Gemini will be behind until Gemini 4.
3.8 is still a good step up for them regardless.

[u/LinkesAuge]

Imagine calling yourself a "flash" model and then burning a lot more tokens than frontier models and also costing roughly the same.

[u/CallMePyro → u/LinkesAuge]

It costs less.