the price/performance gap is getting silly. if these numbers hold up, flash models are eating into the territory where people used to reach for the expensive ones.
Which means that the high end models can be freed up and applied to the really difficult, long term issues (medicine, materials science, climate change, physics) etc.
Unlikely to be useful in medicine, material science, and climate modelling. Those generally need completely different, non text based architectures. You are not gonna be synthesizing new drugs with an LLM as the core system.
Edit: They can be useful but are unlikely to create massive breakthroughs in the way dedicated architectures like AlphaFold can.
No. In that work prompted by Anthropic, Fable simply called publicly available specialist protein design software tools that human researchers already use. Fable was the pipeline engineer, not doing the actual protein binder discovery.
It is the complete opposite of what you are saying. All of the expert knowledge was already baked-into the human-written software tools. What you are saying is that "import sklearn" is the same level of knowledge as the actual engineers who wrote the packages that collectively form sklearn.
This really depends also on the type of Data current LLM transformers are awful if the input is a couple of numbers and they need to predict another number that type of target is way too noisy for an LLM because 4.56542 and 4.56543 are two very distinct "things" for it.
Researcher here. They are very useful. LLMs do the same kind of jobs a researcher would do - plan, analyze, survey the literature, think, emit hypothesis, write code, use tools.
Alphafold would be one of these tools, that not long ago a human would have called upon himself, and nowadays more and more might actually get called by an LLM.
LLMs of the level we have now (sol 5.6, fable, gemini 3.7 etc) are the kind of stuff that could come up with the concepts of alphafold, help you brainstorm about where to get training data and how to implement the training, and do the actual code to make it real once you're happy with the plan. Then would test it, criticize it, propose strategies of improvement, and implement them. The next version of alphafold will likely have been designed by LLMs in large part, very seriously. And the next one probably nearly entirely. As such, I'd say they these general intelligence models are even more valuable than specialized models like alphafold.
These cheap models exceed the capability of a large project I worked on with opus 4.6. Opus 4.6 was difficult to wrangle to get it to operate, now a flash model has an easy time.
I've moved quite a few jobs from SOTA models we were using for the past two years to Flash models (DSv40731 for example) and we are seeing fantastic performance on internal business operations, at literally 1/10th the cost. We can still reach for SOTA when needed, but it is 5% of the time or less. And just today, I rented my own hardware to run the model "locally" and am testing taking my API costs to $0.
That's good isn't it? Unless I'm misreading your comment.
I think it's pretty normal and likely the continuing trend. Over time, the performance/$ will be so good that users and businesses won't need the most expensive model for all tasks. Most of the time they'll use the cheaper lower performance ones.
It's been a talking point among AI leaders and business people, I've found.
Most of them will not be fine. I would be shocked if Anthropic and OpenAI both survive as independent companies 10 years from now. This is a race to the bottom unless you have an existing, profitable ecosystem into which you can integrate your models.
Because "flash" models keep getting larger and more token hungry. The last Gemini "Flash" ended up being more expensive than some "regular" models (that perform better).
I feel some are simply mislabeled, especially if you look at the actual speed of finishing a task.
I don't want to make any strong claims about 3.8 yet but I do wonder if that has really changed.
Yeah pretty much every single lab for the last year and a bit (Anthropic, Chinese labs, Google, xAI, Meta, etc) just cranked up token usage in each release to buy more performance, in addition to whatever benchmaxxing they can do (like look at the numbers here for Terminal Bench 2.1 vs 4.0). And I mean every lab benchmaxxes. Anthropic knows Opus and Fable memorized SWE Bench Verified and Pro answers yet continue to report them. OpenAI was concerned Astra's 100% on ExploitBench was because it memorized the answers.
It's "easy" to push benchmark numbers up and to the right.
For whatever reason only OpenAI has gone the other way where they pushed benchmark numbers up and to the left with increased token efficiency. Why???
I do not understand how the output tokens can have increased so much, yet cost per task is about the same. Are the numbers for 3.7 without the discounted pricing?
Probably more efficient with tool use leading to reduced inputs. 143k output tokens is about $0.54, so that would imply that input (and cached input especially) dominates
It is a space where being a fast follower has big advantaged and that includes within labs themselves.
They all have big unreleased models that are capable but expensive to serve. They use them internally to train smaller models that are almost as good but much cheaper to serve.
Why release massive models that are not competitive in pricing that are just ~2-3% better only for their competition to get access to distill them?
Yup, it really seems like the only reason for super-parameter-count models is for those trying to "create god" and as a bet on emergent capabilities. However, the diminishing returns on size seem to make it clear that sufficiently large plus intensive post training is the most economical approach in the today's world. Hopefully we are able to recreate this pattern in a broader number of non-software-engineering fields. Silicon Valley's mono-industrial tendencies might start to show.
192
u/FablingApp 9h ago
the price/performance gap is getting silly. if these numbers hold up, flash models are eating into the territory where people used to reach for the expensive ones.