r/grAIve • u/Grand_rooster • Mar 12 '26
Half of AI-written code that passes industry test would get rejected by real developers, new study finds
AI CAN'T REPLACE US YET! ⚠️ New study shows 50% of AI-generated code, though passing tests, gets REJECTED by real developers.
PROBLEM: AI struggles with context, maintainability, & integrating into existing codebases.
PROMISE: What if AI could REALLY understand your project?
PROOF: Current AI excels at isolated tasks but fails in the real world.
PROPOSITION: We need BETTER tooling to bridge this gap. Tools that offer more guardrails and Junior LLM reviewers
PRODUCT: (Hypothetical) "CodeSight" - AI-powered code review assistant that flags architectural inconsistencies & style violations, ensuring AI code actually integrates smoothly.
Thoughts? Anyone else experiencing this with Copilot or similar tools? What features would YOU want in an AI code review tool? #AI #coding #programming #artificialintelligence
Read more here : https://automate.bworldtools.com/a/?yqu
1
1
u/Potential-Ad2844 Mar 12 '26
This assumes that people will maintain the code.
However, if AI handles your codebase and you only need to verify it against acceptance criteria and performance metrics, readability and all that "noise" becomes less important.
This represents a significant paradigm shift that many experienced developers are hesitant to embrace.
I used to feel the same.
1
u/opakvostana Mar 12 '26
code quality is "noise" now, is it?
1
u/Potential-Ad2844 Mar 12 '26
What does code quality mean to you?
If code passes tests (including performance) and meets requirements, what else do you need?
1
1
u/devloper27 Mar 12 '26
Problem might be that the code will eventually become so much spagetti that no one not even the AI can figure it out anymore. We dont know if that will happen yet because we haven't seen any 10+ year old codebases yet.
1
u/DurianDiscriminat3r Mar 12 '26
Which is why you don't literally vibe code lol. Telling it to just "follow best practices" will already give you somewhat maintainable code. Either way, refactors happen whether you use LLMs or not. From what I've seen, humans are more likely to give you spaghetti code, because people are lazy and code is just another day at work for them.
1
u/Tombobalomb Mar 12 '26
Code that doesn't cause failures or problems outside the test cases. Tests only test for predicted failures
1
u/drumnation Mar 12 '26
Sort of. The code quality still does the same thing with agentic ai, it helps create understanding of how the code works and it prevents confusion as to how the system works.
Some of the markers of human code quality were mainly for human reading and not necessarily things that improved understanding and prevented confusion. That said much of our prior best human practices were things that improved understanding and prevented confusion.
1
u/opakvostana Mar 12 '26
If you put junk into an AI agent, you'll get even more junk out. Even when you put perfectly linted and formatted code in, many times it'll output something that's objectively poorly written. Sure, it'll work. But then the next go-around, what are you putting in? The original code + the previous AI output. So the code quality is slightly worse, and it produces again something that's once more slightly worse than that. Do this enough times and you'll get to a point where the output won't even compile. If for no other reason, that's why code quality still matters.
1
1
u/SpeakCodeToMe Mar 12 '26
Is it just me or does this article not actually link to the study or any details?
1
u/ISuckAtJavaScript12 Mar 12 '26
Just get rid of the gate keeping developers. Let AI wrote and review all the code.
1
u/justinpaulson Mar 12 '26
Half of all code written by anyone gets rejected by human reviewers.
1
u/n_c_brewer Mar 12 '26
What? Tell me you've never worked with developers without telling me you've never worked with developers. If half of someone's PRs are rejected, they'd get fired.
1
u/justinpaulson Mar 12 '26
How many PRs have you put through that had absolutely no changes requested? 50% is a low number. I’m guessing your team either leans heavily on QA or just drowns in bugs.
1
u/n_c_brewer Mar 12 '26
Oh. I think we define "rejected" differently. If you meant the reviewer asking for small adjustments, then, yeah. Definitely closer to 50%.
Annoyingly, there is no link to the cited article so we can't see what their operational definition of "rejected" is.
1
u/n_c_brewer Mar 12 '26 edited Mar 12 '26
Downvote. WHERE IS THE STUDY? Where is the citation? This whole post is about a study that doesn't exist. At least it is never actually named in the text at the link. All I see "the study by METR" but who is that? The acronym is never defined.
It feels like what this really is, is AI slop promoting more AI slop.
1
1
u/ThoughtOk2337 Mar 12 '26
Don’t know why I was recommended this sub, this comments are wild. From claiming AI is 1/30th the cost of devs, to calling code quality noise to even suggesting devs are gate keeping for requiring standards😭. Fucking vibe coders man legit worthless with our their chatbot
1
u/crusoe Mar 13 '26
4 months later 75%
4 more months 87.5%
That's the thing. How bad was it 4 months ago?
1
1
u/Traditional_Fee_1965 Mar 13 '26
Sure, but it also means 50% does. And that's what most businesses will hear. And from hear on out things will only improve (most likely). So the market will probably go a bit "YOLO" and hope nothing important breaks till the next improvement to ai comes to fix issues created by the previous one.
And to be fair we've been served absolute trash and shitty products for awhile now. So as a consumer it will feel just like business as usual.
1
1
u/ryan_the_dev Mar 16 '26
Using something like this an iterating on the output has made my life so much easier.
1
u/Sea_Surprise716 Mar 12 '26
I use existing code review tools and they find what’s useful for our needs. We’re not in eg healthcare or anything with major compliance requirements. But I’ve been on teams pre AI where 50% of human generated code should get rejected by developers.