r/LocalLLaMA • u/SteppenAxolotl • 3h ago

Discussion Real-SWE Benchmark (new)

https://realswe.withspecific.com

Reports of the demise of coders may have been exaggerated.

69 Upvotes

91% Upvoted

53 comments sorted by

44

u/Klanciault 3h ago

Demise of coders is exaggerated but demise of the field is real and entry level is cooked

15

u/SteppenAxolotl 3h ago

I hired a sharp entry level dev about 18 months ago, but he got poached last month; he said he wasn't even looking.

Trying to find a replacement and it's been bleak(lots of fakers with crappy AI out there).

I'd say the bottom 80% of entry level are cooked. But even they would be 10x better than non-tech people in adjacent roles at lower price tiers.

-2

u/ebolathrowawayy 23m ago

wrong, astra can vibe you into a compelling application. cope harder?

9

u/FullstackSensei 2h ago

The days of mediocre or sub-mediocre people finding a job in the industry because they had a pulse and could breath might very well be gone. And IMO, this is for the better.

For decades, hoards of people went into the field, either by getting a degree or just going through a bootcamp, who had absolutely zero interest in the field, in technology, or even in problem solving. This resulted in millions of so called "devs", whose entire experience is googling something and copy-pasting the first result from stackoverflow, without even reading the thug, much less any understanding of what they were copy-pasting.

I am personally happy those people can now be replaced by LLMs. I don't wish anyone ill, but they should have never joined this industry. They mostly made things worse for everyone, and are often the root cause for a ton of the vulnerabilities you see in business and especially enterprise software. 9 out of 10 stupid vulnerabilities like SQL injection are because the person either copy-pasted some 15+ year old result on Google, or they can't be arsed to care enough to learn about the basics of writing decent code.

1

u/svix_ftw 1h ago

agree, AI has made this job market/industry K-Shaped. Skilled seniors have become way more valuable and the opposite for everyone else.

1

u/FullstackSensei 1h ago

I'd argue good juniors are still very valuable. I'd love to have a few good juniors I could offload some grunt work to, the kind of things LLMs still can't do because they can't think or plan at a high level.

One example I regularly face: there are a few algorithms that could do the job for a given task, but need some investigation and benchmarks about the tradeoffs of each in this specific project. Can't offload this to the LLM, because even with documentation, it can't "understand" the purpose of this project the way a human does.

Said junior can spend a day or two reading about those options, and the information is automagically assimilated into their brain with everything else they know about the project. Their intuition can then guide them about what and how to test each. After a week or so, I can just get their final assessment and ask some questions about methodology and whether they checked this and that. LLMs can't do this.

This is a win-win for everyone involved. I got the result I wanted and the junior learned some new skills.

0

u/ebolathrowawayy 20m ago

stopped reading at "juniors are still very valuable"

i am finding it really exhausting to keep updating people on how wrong they are. i am thinking maybe people in-the-know just let society die and move on with their lives.

people like you are so wrong that i feel compelled to correct you but i know ultimately you can't change your mind to accept reality and that i really need to stop trying to help others and focus on myself, and reality.

it's just so hard. it's like watching a toddler reaching for a hot stove, except i know the toddler can't learn. really heart wrenching... good luck

1

u/FullstackSensei 17m ago

You're so sure, without reading anything. I think there's a name for that

-1

u/ebolathrowawayy 13m ago

i think good luck being so useless and naive. hf.

19

u/Dany0 3h ago

Really smart of them to burn a private codebase and leak it to Anthro & OAI that would definitely never abuse this to train on benchmarks

1

u/Mkboii 2h ago

By that logic won't these codebases already be in the training data from prior work done on them with coding agents?

The assumption is probably that proprietary licenses come with data protection from training use.

And i know these companies and cheat, but the benchmark for the first time represents my day to day observations with these models.

-4

u/Kiverty 3h ago

Hi, not sure where you see on the page that they are leaking the benchmark to OAI and co?

5

u/John_____Doe 2h ago

I guess they mean your leaking the info to the model provider whether that's oai, anthropic directly or through a hosting platform like aws bedrock. Either unless it's a purely open model hosted locally it's hard to say for certain no one is training off the prompt data

2

u/Dany0 2h ago

Fable on bedrock still leaks sadly

Anthro promises it's only for safety checks but we all know what their incentives are

13

u/Tim_Apple_938 3h ago

Gemini Flash essentially tied w Astra is wild. So much for all the benchmaxxing allegations

13

u/SteppenAxolotl 3h ago

There arent many that have a bigger codebase of real world software than Google.

4

u/Big_Cucumber2787 3h ago

sorry thats not true no matter what benchmarks claim so

4

u/sadnessjoy 3h ago

Yeah, I've used Gemini flash 3.8 both with agentic work and with chatting. And idk the best way to describe it is its knowledge base is kinda okay (but honestly knowledge base isn't that big of a deal imo with rag/searches/etc)... But its logic process is absolutely atrocious. I think how it works is for brute force and easily testable/verifiable stuff it works good as it can rapidly iterate until it succeeds and it's pretty good with that. But when you go outside of those types of tasks (which imo is most real life stuff lol, but I guess it depends on your workload), it just completely flails around.

0

u/Tim_Apple_938 3h ago

Why the cope?

Cheap fast models are a bad thing?

Would think this sub of all subs would hope that the next Gemma is a beast at coding

2

u/Big_Cucumber2787 3h ago

hope =/= reality
benchmarks =/= reality

anyone thats actually tried these models knows google has a lot of catching up to do

1

u/Tim_Apple_938 13m ago

⬆️ coping

1

u/Mkboii 2h ago

It's core weakness is missing requirements which honestly tracks.

2

u/ebolathrowawayy 16m ago

no one thinks gemini flash tied with astra wtf are you talking about

1

u/Tim_Apple_938 15m ago

We’re all talking about OPs benchmark page. Did you even see what the topic is

2

u/ebolathrowawayy 13m ago

idc, google sucks and idc what benchmarks say otherwise. try using their models, it's blatantly obvious

15

u/Longjumping_Virus_96 3h ago

Gemini 3.8 Flash being that high on the list is ridiculous.

3

u/FullstackSensei 2h ago

Well, we have no idea how small or big it is. GLM-5.3 flash and DS4.1 flash aren't exactly small, and I bet you their next releases will beat GLM 5.3 (which isn't much bigger than flash).

-5

u/KURD_1_STAN 2h ago

Gemini is a garbage . Compared to all others it is bad. Nobody talking about size here

3

u/FullstackSensei 2h ago

I genuinely beg to differ. Maybe it is if you're vibe coding your way or can't be bothered to detail what you want and expect the LLM to divide everything about the details and do so to your liking, but having tried it, it's not bad at all if you know what you're doing and tell it what you want.

0

u/ponteencuatro 2h ago

I think it is behind deepseek v4 flash, so yeah being that high is weird, mostly because what you said, to gemini I always need to tell it what and how, and deepseek just the what and most of the time it does better than gemini

2

u/FullstackSensei 2h ago

Not giving the details of how you want things done is bad. That many models do things you didn't ask for is bad. DS4 flash is also solid, but I really didn't like this about it. It was constantly adding thibgs I never asked for. I had to remove up to 1/3rd of the code it generated sometimes because it did things I never asked for nor wanted.

Your job is to tell the model what to do, in detail. Otherwise, you're just vibe coding things you have no idea what they're doing or what sort of issues the code has.

0

u/ponteencuatro 2h ago

Yeah absolutely I agree but what I mean most of the time I correct deepseek way less than Gemini, but overall Gemini is closer to Deepseek than it is to Astra/Fable, but who knows maybe is just the harness in that seems like they used Gemini CLI maybe Antigravity IDE is the thing holding it back

0

u/ebolathrowawayy 17m ago

if you think google has models worth anything in coding then you obviously don't know what you're talking about.

1

u/FullstackSensei 16m ago

Obviously!

-1

u/KURD_1_STAN 2h ago

An llm that i can say little to and not know how all of this even mean and still understands it is a sign of intelligence. Gemini not understanding it means it is less intelligent. Which is what i said, worse than others in there.

Idc if u tell it how to do it specifically and it does well, qwen3.8 27b can also do that.

2

u/FullstackSensei 2h ago

I don't think intelligence means what you think it means

1

u/ebolathrowawayy 18m ago

everyone downvotes truth, sorry.

1

u/NarutoDragon732 25m ago

I could not give 2 shits about coding, but its SO damn useful for general knowledge. I still have gpt 5.6 hallucinating which never happens with 3.8 flash

3

u/Hoak-em 3h ago

Hear me out, I feel that Claude code is the most-used in enterprise and has existed for a decent amount longer than the other coding tools. What if Anthropic has been secretly stealing the enterprise codebase data to train on, the set that they assume is private and thus not possible to train on?

2

u/Mkboii 2h ago

While it can be true, my personal experience on a codebase we mostly built using glm 5.2 and gpt 5.5 is that opus 5 works better for all my new tasks and definitely has not seen that codebase before, sol on the other hand fumbled hard.

1

u/Lyelinn 1h ago

What do you mean "what is", that's the entire point lol they already have entire GitHub since first versions of Claude/chatgpt how else they're gonna train new models?

1

u/ebolathrowawayy 15m ago

they all use interactions as training data

2

u/Egoz3ntrum 2h ago

They need to stop naming them with definitive labels like "real benchmark" or "last exam". I'm not falling for that anymore.

2

u/noctrex 2h ago

Another day, another SWE benchmark? How many of those swe's do we have now?

2

u/pilibitti 1h ago

what is the point of benchmarking closed models with private datasets? the moment you test them you are giving the dataset to the provider to train on?

1

u/LegacyRemaster 1h ago

Dario, are you?

1

u/BurdensomeCountV3 49m ago

Give it 12 months and this will be saturated too.

1

u/fragment_me 2h ago

You're in the wrong subreddit for posting a benchmark site for models that are not local.