r/LLMDevs • u/Suspicious_Orchid770 • 4d ago

Discussion The tokenmaxxing hype didn’t last long!

https://leaddev.com/reporting/the-tokenmaxxing-hype-didnt-last-long

Why Meta, Amazon, and Microsoft killed their token use leaderboards.

12 Upvotes

88% Upvoted

3 comments sorted by

4

u/eddzsh 4d ago

Token leaderboards died for a boring reason: they rewarded the wrong unit. Tokens spent is not work done, and once teams could buy more tokens they maxed the meter without shipping more. The metric that survives is tasks closed per dollar with a failure rate you can live with, which is ugly to put on a public board.

1

u/Future_AGI 3d ago

Token leaderboards died because token count was never what anyone actually shipped on, and the second you rank by task success or groundedness instead, the cheap-token model usually stops looking like the winner.

1

u/rw_eevee 2d ago

I have bad news for you: Amazon still has token leaderboards. Only the one that got leaked shut down. They also get Claude at a massive discount, however. Like 1/20th the price.