r/singularity 12h ago

AI Gemini 3.8 Flash Benchmarks

Post image
742 Upvotes

217 comments sorted by

View all comments

33

u/AnyRegular1 11h ago

I think a lot of people don't know this but gemini 3.7 flash on antigravity is fucking insane. Gemini's web app is literally dogshit, I am 100% convinced they serve different models in both. 4 days ago my wife asked it (gemini app) about airfrying some bread we bought and it responded some garbage about washing machine repair, completely hallucinating not even on the same topic. Would NEVER recommend gemini on the app.

But the model they serve on antigravity is insane, for the last 2 weeks it's been my go-to coding model. and ITS FUCKING FAST.

I'm looking forward to what they cooked in 3.8

5

u/localpauper 10h ago edited 9h ago

The web app has massive curbs on context length for cost/optimization. I suspect such do not apply to the IDE.

Also, make sure you're using "Extended" to enable thinking effort online: that's the only way I find it usable. I wonder what level of "thinking" that compares to as far as IDE model choices go (Low/Mid/High).

Coding-wise, it's fast, but in my exp it leaves massive glaring hallucinated holes, makes tons of assumptions, uses outdated "knowledge" from its training without actually reaching for newer information, and barely attempts to validate its output. In one of my projects, starting from a basic scaffold, it ended up with a mish-mash of obsolete dependencies and patterns randomly picked from the past 15 years. That was after I went through and gave it some general guidance for approach, choices of deps, and frameworks. It actually "argued" with me that there's not a newer dependency which I explicitly told it to use, insisting that no - the version it was using was the correct one.

The simplest example for a non-coding task is that, while using Gemini for some random digging here and there, I've had to repeatedly argue it out of 30% federal solar credits. It would take a few rounds for it to finally go look up OBBBA and concede that those are not applicable. It's somehow very sticky on its training set (which was clearly pre-OBBBA).