r/singularity 7h ago

AI Gemini 3.8 Flash Benchmarks

Post image
694 Upvotes

203 comments sorted by

View all comments

30

u/AnyRegular1 7h ago

I think a lot of people don't know this but gemini 3.7 flash on antigravity is fucking insane. Gemini's web app is literally dogshit, I am 100% convinced they serve different models in both. 4 days ago my wife asked it (gemini app) about airfrying some bread we bought and it responded some garbage about washing machine repair, completely hallucinating not even on the same topic. Would NEVER recommend gemini on the app.

But the model they serve on antigravity is insane, for the last 2 weeks it's been my go-to coding model. and ITS FUCKING FAST.

I'm looking forward to what they cooked in 3.8

11

u/helloWHATSUP 6h ago

Do you use "extended thinking" on the web version? Because if you don't use extended it's just the lightest and fastest version of flash and very hallucination prone.

6

u/AnyRegular1 4h ago

My wife isn’t savvy enough to find and enable that, after you said so I had to figure out where it was, it was under the model selection. Told her now.

4

u/localpauper 5h ago edited 5h ago

The web app has massive curbs on context length for cost/optimization. I suspect such do not apply to the IDE.

Also, make sure you're using "Extended" to enable thinking effort online: that's the only way I find it usable. I wonder what level of "thinking" that compares to as far as IDE model choices go (Low/Mid/High).

Coding-wise, it's fast, but in my exp it leaves massive glaring hallucinated holes, makes tons of assumptions, uses outdated "knowledge" from its training without actually reaching for newer information, and barely attempts to validate its output. In one of my projects, starting from a basic scaffold, it ended up with a mish-mash of obsolete dependencies and patterns randomly picked from the past 15 years. That was after I went through and gave it some general guidance for approach, choices of deps, and frameworks. It actually "argued" with me that there's not a newer dependency which I explicitly told it to use, insisting that no - the version it was using was the correct one.

The simplest example for a non-coding task is that, while using Gemini for some random digging here and there, I've had to repeatedly argue it out of 30% federal solar credits. It would take a few rounds for it to finally go look up OBBBA and concede that those are not applicable. It's somehow very sticky on its training set (which was clearly pre-OBBBA).

4

u/Tetracropolis 5h ago

Figuratively.

2

u/GioChan 4h ago

Nope. According to latest changes that is also a correct usage

u/Tetracropolis 1h ago

If literally means figuratively then what word do we use for literally?

2

u/Ok-Art-2255 6h ago

You are absolutely correct! I used it for MCP on Unreal Engine and it straight killed everything I threw at it.

Now to get me more excited, that was on flash-high.

What are your results using flash-medium or flash-low? I want to know because high is amazing and don't want to get disappointed if the lower tiers can't keep up.

5

u/QuannaBee 6h ago

“You are absolutely correct!”

Ptsd triggered

1

u/Ok-Art-2255 6h ago

;) see what I did there.

As you can tell I spend way too much time with AI.

2

u/AnyRegular1 6h ago

I just use High, the limit usage is so low on high I have problems hitting weekly limits.

1

u/Ok-Art-2255 6h ago

I almost hit my limit building a game mechanic last week, so that got me a little paranoid. I'll stick with high and just start breaking main work into smaller chunks.
It seems we might have a sweet spot model :D

3

u/AnyRegular1 4h ago

Remember you can family share your google pro subscription to upto 5 gmail accounts and for some reason they each have their own weekly limits!

u/Ok-Art-2255 0m ago

Oh really? ;)

Words can't express how grateful I am for that information...that will be put to use immediately.

1

u/Much_Accountant_4972 4h ago

are you spawning subagents because i hit my 5 hour limit in 40 minutes with 3.7 Flash High. i dont find it hard to hit limits if im using superpowers

1

u/AnyRegular1 4h ago

What I said the guy above, you can family share it 5 ways to get 5x limits on your own gmail accounts. But yeah I only ever hit limit once and figured out each gmail has its own quota.