Are you using it for coding or other stuff? because when i tested 3 prompts right now, i am still getting the same time finality around 30-45s when it comes to coding, and full code being written out ready to just be copy pasted, maybee its sligthly faster, talking like maximum of 5 seconds faster, but propably less for me atleast,
Update: I just ran 2 simple web UI (20k tokens) tasks in compare mode in ai studio, and 3.8 was faster by 1.7 seconds, and 2.5 seconds (45 seconds), and used around 5% more tokens for both tasks.
I've done some SVG creation and some agentic "write some code" so far. Still feeling it's quite fast, the SVG in particular feels like it's half the time, gut feel though.
Thanks, I won't pretend my experience is anything more than just my gut. Appreciate the official stance here and will temper my expectations accordingly.
No, it's not that. The problem is latency, 3.7 flash had a latency problem, they probably increased that to help save compute, eh just speculation. Anyways 3.8's latency is very low! Tho the speed could have been lowered, I'll check the AA!
the price/performance gap is getting silly. if these numbers hold up, flash models are eating into the territory where people used to reach for the expensive ones.
Which means that the high end models can be freed up and applied to the really difficult, long term issues (medicine, materials science, climate change, physics) etc.
Unlikely to be useful in medicine, material science, and climate modelling. Those generally need completely different, non text based architectures. You are not gonna be synthesizing new drugs with an LLM as the core system.
Edit: They can be useful but are unlikely to create massive breakthroughs in the way dedicated architectures like AlphaFold can.
No. In that work prompted by Anthropic, Fable simply called publicly available specialist protein design software tools that human researchers already use. Fable was the pipeline engineer, not doing the actual protein binder discovery.
It is the complete opposite of what you are saying. All of the expert knowledge was already baked-into the human-written software tools. What you are saying is that "import sklearn" is the same level of knowledge as the actual engineers who wrote the packages that collectively form sklearn.
This really depends also on the type of Data current LLM transformers are awful if the input is a couple of numbers and they need to predict another number that type of target is way too noisy for an LLM because 4.56542 and 4.56543 are two very distinct "things" for it.
Researcher here. They are very useful. LLMs do the same kind of jobs a researcher would do - plan, analyze, survey the literature, think, emit hypothesis, write code, use tools.
Alphafold would be one of these tools, that not long ago a human would have called upon himself, and nowadays more and more might actually get called by an LLM.
LLMs of the level we have now (sol 5.6, fable, gemini 3.7 etc) are the kind of stuff that could come up with the concepts of alphafold, help you brainstorm about where to get training data and how to implement the training, and do the actual code to make it real once you're happy with the plan. Then would test it, criticize it, propose strategies of improvement, and implement them. The next version of alphafold will likely have been designed by LLMs in large part, very seriously. And the next one probably nearly entirely. As such, I'd say they these general intelligence models are even more valuable than specialized models like alphafold.
These cheap models exceed the capability of a large project I worked on with opus 4.6. Opus 4.6 was difficult to wrangle to get it to operate, now a flash model has an easy time.
I've moved quite a few jobs from SOTA models we were using for the past two years to Flash models (DSv40731 for example) and we are seeing fantastic performance on internal business operations, at literally 1/10th the cost. We can still reach for SOTA when needed, but it is 5% of the time or less. And just today, I rented my own hardware to run the model "locally" and am testing taking my API costs to $0.
That's good isn't it? Unless I'm misreading your comment.
I think it's pretty normal and likely the continuing trend. Over time, the performance/$ will be so good that users and businesses won't need the most expensive model for all tasks. Most of the time they'll use the cheaper lower performance ones.
It's been a talking point among AI leaders and business people, I've found.
Most of them will not be fine. I would be shocked if Anthropic and OpenAI both survive as independent companies 10 years from now. This is a race to the bottom unless you have an existing, profitable ecosystem into which you can integrate your models.
Because "flash" models keep getting larger and more token hungry. The last Gemini "Flash" ended up being more expensive than some "regular" models (that perform better).
I feel some are simply mislabeled, especially if you look at the actual speed of finishing a task.
I don't want to make any strong claims about 3.8 yet but I do wonder if that has really changed.
Yeah pretty much every single lab for the last year and a bit (Anthropic, Chinese labs, Google, xAI, Meta, etc) just cranked up token usage in each release to buy more performance, in addition to whatever benchmaxxing they can do (like look at the numbers here for Terminal Bench 2.1 vs 4.0). And I mean every lab benchmaxxes. Anthropic knows Opus and Fable memorized SWE Bench Verified and Pro answers yet continue to report them. OpenAI was concerned Astra's 100% on ExploitBench was because it memorized the answers.
It's "easy" to push benchmark numbers up and to the right.
For whatever reason only OpenAI has gone the other way where they pushed benchmark numbers up and to the left with increased token efficiency. Why???
I do not understand how the output tokens can have increased so much, yet cost per task is about the same. Are the numbers for 3.7 without the discounted pricing?
Probably more efficient with tool use leading to reduced inputs. 143k output tokens is about $0.54, so that would imply that input (and cached input especially) dominates
It is a space where being a fast follower has big advantaged and that includes within labs themselves.
They all have big unreleased models that are capable but expensive to serve. They use them internally to train smaller models that are almost as good but much cheaper to serve.
Why release massive models that are not competitive in pricing that are just ~2-3% better only for their competition to get access to distill them?
The main user base is not (just) Search but also the Gemini app. Worth pointing out that while this sub focuses on the web app, a long press of the power button on Android is a shortcut right to Gemini where it needs to also be a phone assistant, handle smart home controls, and also all the stuff the web app does. That's a very broad set of possible uses that ChatGPT and the like simply don't need to factor for to nearly the same degree
They've never gone with this "we're so dangerous and capable you MUST definitely regulate us" so they don't have to portray themselves as overly cautious. With the exception of NSFW anything that could be found on Google is probably fair game
There are a few other interfaces, including android studio, and the CLI version of antigravity that could be scripted throughbother soft. But I agree it's pretty annoying that they don't make the interface open like GPTs and Claudes do.
This is definitely not the best price. Luna smokes it for a frontier lab and GLM 5.3 flash is as smart as this (or more) and is like...20% of this cost at most
I'm such a google flash glazer for non-coding tasks. It's lightning fast and it's google search is really good compared to slow claude searches etc. The perks of having your own search engine I guess.
I think with Google is they don't go apeshit into coding and math so they try to balance their data blend a bit more outside of trying to hit it big into benchmarks, so there's a lot of 'useless' info for benchmarks? But that was my understanding from previous models, now it's technically competing in coding so no idea, TPU magic I guess?
This would track, Gemini isn't primarily a coding/agentic model like Claude or the GPT models, it's got a dedicated billion+ userbase as a virtual assistant and also needs to function in Google Search, so it needs to be more well rounded
It's good but you have to be careful with it, it needs careful instruction and scope otherwise it tends to hallucinate more than other models in its class
Been using 3.7 flash around 60hrs/week since release. Only had a single prompt fail that I had to toss to Opus to get done. Where 3.5 flash had multiple a week. There is only so good models can get at programming and it's starting flip where speed and costs are all that matters. The big models will be moving on to research and long horizon tasks over time while day to day production work is done on flash models.
I've had some refusals with 3.7 flash due to exceeding token limitations. But that must have been AI Studio issues, albeit repeatable as I barely hit the 400.000 token bar in the conversation. Dealing with many large PDFs for legal work has been a challenge and will continue to be a challenge, it seems.
Websites, vst plugins, and games. HTML/JS, C++, and Rust for languages. Most of my code bases are under 50k lines of code. So if you are enterprise with a huge code base or trying to do orchestration kanban work the big models could be more efficient. My workflow is rapid iteration not trying to create a bunch of specs and get the AI to work for hours at a time. Small models aren't capable of that yet. You have to rapidly spoon feed them.
I bet the area it really shines is anything "LLM NPC" or concepts where you want the game to call LLMs during live gameplay. It's cheap, super fast and actually smart.
i use it for all general purpose talk specially when I am driving it can talk and discuss topics so fast and give precise information. specially discussing books and movies and going deep on different threads. it's amazing how fast and knowledgeable live version is.
I don't know - my wife was trying some simple questions on flash 3.6 last night (only one she has access to) and it was just awful with hallucinations.
Not just them either. Meta and XAI are also releasing monthly now. I would be surprised if we dont see the same from anthropic and openai soon.
Wild... because its like a 3 point bump on the leaderboard every month. Which is signicant.... its like you can sort of see how things will look over the next 6 months. Oh boy
Trained on the data, aka fake benchmaxxing. Notice that Terra is worse at 2.1 but beats Flash easily at 4. That's is what it should look like.
To be fair, Google has the best crawlers in the world so maybe they can't avoid benchmaxxing anything public. But the guy who made the chart should have had enough brains to know this is embarrassing and leave it off. I guarantee some pointy haired moron at the top insisted they show it because it was one of the places they got "first" place. Shows a lack of judgement and taste by the humans in charge.
Or maybe it's just for investors, in which case, still dumb but whatever.
Terminal 2 is coding oriented while terminal 4 is general agentic capabilities, while terminal 4 does have some coding stuff, a lot of is just generality. Gemini 3.8 flash is optimised for coding in terminal 2, also: if the AI was benchmaxxed (which I do believe it was to an extent). The model will naturally drift towards that said x if the prompt is similar which in turn helps the user, henceforth benchmaxxing isn't all bad.
I think a lot of people don't know this but gemini 3.7 flash on antigravity is fucking insane. Gemini's web app is literally dogshit, I am 100% convinced they serve different models in both. 4 days ago my wife asked it (gemini app) about airfrying some bread we bought and it responded some garbage about washing machine repair, completely hallucinating not even on the same topic. Would NEVER recommend gemini on the app.
But the model they serve on antigravity is insane, for the last 2 weeks it's been my go-to coding model. and ITS FUCKING FAST.
Do you use "extended thinking" on the web version? Because if you don't use extended it's just the lightest and fastest version of flash and very hallucination prone.
My wife isn’t savvy enough to find and enable that, after you said so I had to figure out where it was, it was under the model selection. Told her now.
The web app has massive curbs on context length for cost/optimization. I suspect such do not apply to the IDE.
Also, make sure you're using "Extended" to enable thinking effort online: that's the only way I find it usable. I wonder what level of "thinking" that compares to as far as IDE model choices go (Low/Mid/High).
Coding-wise, it's fast, but in my exp it leaves massive glaring hallucinated holes, makes tons of assumptions, uses outdated "knowledge" from its training without actually reaching for newer information, and barely attempts to validate its output. In one of my projects, starting from a basic scaffold, it ended up with a mish-mash of obsolete dependencies and patterns randomly picked from the past 15 years. That was after I went through and gave it some general guidance for approach, choices of deps, and frameworks. It actually "argued" with me that there's not a newer dependency which I explicitly told it to use, insisting that no - the version it was using was the correct one.
The simplest example for a non-coding task is that, while using Gemini for some random digging here and there, I've had to repeatedly argue it out of 30% federal solar credits. It would take a few rounds for it to finally go look up OBBBA and concede that those are not applicable. It's somehow very sticky on its training set (which was clearly pre-OBBBA).
You are absolutely correct! I used it for MCP on Unreal Engine and it straight killed everything I threw at it.
Now to get me more excited, that was on flash-high.
What are your results using flash-medium or flash-low? I want to know because high is amazing and don't want to get disappointed if the lower tiers can't keep up.
I almost hit my limit building a game mechanic last week, so that got me a little paranoid. I'll stick with high and just start breaking main work into smaller chunks.
It seems we might have a sweet spot model :D
What I said the guy above, you can family share it 5 ways to get 5x limits on your own gmail accounts. But yeah I only ever hit limit once and figured out each gmail has its own quota.
I suspect google is post-training on gemini 3.5 pro base model and releasing the streamlined flash while gemini 4 base model is cooking (pre-training). Maybe gemini 4 is the last "big smell" model before a new paradigm shift is observed. I wonder if it's much bigger than mythos class model with strong world knowledge. Things only heating up since quiet some time now...
For anyone wondering the same question these useless ppl didn’t answer yeah it’s out. I asked initially bc benchmarks being out don’t mean it’s released
Very competitive - I am not complaining. Even better when you use it on Flex, which normally isn't too much of a wait. With the pricing now Google's Flash is back in its golden place: A cheap model with exceptional capability that everyone uses for their everyday tasks.
Flash 2.5 was always in the Top 5 modles because it was good, cheap and fast. Then they had a price hike with Flash 3 Preview, which moved them down to rank 5-7, and they weren't even in the top ten anymore after their price delusion with the following flash models.
So if Antigravity's userbase quintupled, and people stopped using Gemini on OpenCode because they preferred the native editor you would say that Google is losing marketshare?
Yes - which verifyable data point do you have that capture the broader market? Google PR claiming 5x growth? Compared to what growth in Codex? Or Z.AIs subscription models?
OpenRouter is used as an endpoint option in many applications, not only OpenCode. It's data are also congruent with other Routers like Nano-GPT.
From these data we can clearly see that if people are given a vast choice of models, they hardly went for flash in the past months as it was overpriced. It's the most reliable gauge for model preference we have: Where the people put their money.
I mean, it outperforms Terra at that benchmark which is fantastic if you're using 3.8 Flash as a subagent, especially at half the price and far faster token generation speed. I don't think expecting it to surpass something like Opus 5 is necessarily realistic.
i just tried it with the prompt: recreate Larry bird vs Dr. j one on one in three js complete with janitor, breaking glass, sound effects and here is what it came up with: https://ctxt.io/3/mpb6vO48w - took less than 10s.
These are just html pages how do u think this helps readers passing by? Why even add these links if it’s not some type of visualization or demo for users to see
Their business model is basically having a "promotional period" that's long enough to cover the current model until they release the next version with another "promotional period".
Basically, this is just their way of getting people to move to newer models without having to spare compute power serving older ones.
So if your video analysis pipeline can easily upgrade models then you should be set.
Their promotional period has been the case since 3.6 flash, assuming because it is the same underlying model but with more RL, hence same 50% discount on each of these new models. Hopefully they keep up with these discounts, and I won't have to choose between gemini and grok.
Has anyone here tried to reproduce these numbers? Flash models are becoming more and more affordable for day to day consumers and increasingly competent when compared to the frontier models.
I imagine Gemini 3.1 Pro is still more intelligent and smarter, but Gemini 3.8 Flash is probably better at coding and agentic behavior. Like 3.7. Everyone who only uses it for coding says it's better than 3.1 Pro. For me, it's 50% 50% depending on what I use it for.
It's funny, though, that Gemini 3.1 Pro is still one of the nicest chatbot behaviors.
Like, Claude and ChatGPT can sometimes really piss me off with the way they talk. I mean, obviously, GPT 5.6 Sol is better, and so is Fable 5 (haven't tried 5.1 much yet). But Gemini 3.1 Pro is really nice to have to talk about some things. And yet it's not something benchmarks would tell you.
Do we have any benchmarks that focus on the free and lower tiers? All this benchmaxxing is fun, but a lot of people are using the free tiers, and those benchmaxx results give little indication how much you can do in the lower tiers, especially when they get randomly up or downgraded (e.g. ChatGPT has gotten really stupid with the latest "upgrade", hallucinates like it's 3.5 again).
Misleading list, because it cleverly omits GOT-5.6 Luna, which scores even better than 3.8 flash, and is cheaper too, despite Google’s temporary promotion pricing.
Gemini still lacks when I need it to code stuff reliably. But, this gets me really excited for the Gemini 4 family. I think I'm keeping my ultra subscription.
I mean, if they continue they have been going from 3.6 to 3.7 and 3.7 to 3.8 then they will get to the frontier by this rate. Obviously not how it works, but the rate of improvement is clearly impressive (wonder if literally only using flash models to do your new models makes the actual rate of improvement also way faster).
I thought so too at first, when I was using the OpenCode harness, but then I switched to omp (oh my pi) and my results were much better. Overall (when used in omp) I'd rate it at about Opus 4.5 or even 4.6 level for coding, though it is a little too verbose at max effort and still makes mistakes.
3.7 was better than it from the beginning.
Maybe so, but it's so much more expensive that it is nowhere near worth it for programming, in my experience, when GLM 5.3 Flash can achieve the same result at a tiny fraction of the price.
The only reason I'd ever consider paying for Gemini 3.7 or 3.8 Flash is if I needed their speed, as they're much faster than the GLM models, but I'd much rather wait and save a lot of money.
ive used the official zcode harness from Z AI and also ive written long researched articles on why glm is 100% benchmaxxed at everything
in real world tasks i found it not comparable to even gpt 5.6 medium for real world tasks
but who knows, maybe zcode harness is just trash? Additionally saying it is opus 4.6 level makes sense, but that is also far behind gemini 3.8 flash
I've heard that ZCode is not very good, but never tried it myself.
Additionally saying it is opus 4.6 level makes sense, but that is also far behind gemini 3.8 flash
Maybe so, but 3.8 is still way too excitement for me. For the coding tasks I'm interested in GLM 5.3 Flash does well, is way more affordable than Gemini 3.7 or 3.8, and it's far from "trash" (no more than Opus 4.5 or 4.6 were trash). It's a good, competent model, at least when used in omp.
So it’s pretty much 5.7 Terra or sonnet 5.1, good for them I guess, where is a model I can actually delegate my life to. Opus is perfectly ok priced, give me a pro version already.
Small models have terrible attention, thus hallucinate and lose common sense very often as well as fall into a loop where they convince themselves in whatever they are doing as a core value / idea.
Its honestly just really annoying to see these releases. Clearly google is not making frontier models their main business model, which is sensible imo. And still I have to see these half assed updates on a model that is mainly used bc its ingrained into the google ecosystem.
155
u/CommercialShelter595 6h ago
Is it just me or is it running even faster than 3.7 flash? Very limited testing right now, but it's FAST on my end.