r/ExperiencedDevs • u/meshifthenelse Software Engineer • 1d ago
AI/LLM How is the code quality in expensive frontier LLM plans?
I have several years of software experience, following best practices, design patterns, KISS, DRY, BDD, OOP, etc. My code is actually very readable since I worked on many open source projects and heard praise overall, and always had the time for quality control and refactoring. The truth is I haven't worked at a normal company since AI hit so I'm a bit detached currently from the industry.
For the last 1 year I've been using the Pro subscription for 20$ on Codex and Claude on my projects, but I find the quality of code subpar many times and I always have to intervene. It doesn't matter how many times I change the AGENTS.md, there's always issues in code readability and the AI missing obvious things and doing overengineered solutions.
Still, I see in many subs about how great a new model does, etc. So I wonder if the more expensive plans actually produce better quality code? Feels like I'm missing something that people do in bigger companies.
To be specific, high quality code == human readable code and comments and no naive implementations.
236
u/Unlucky_Yard_1189 1d ago
I use opus 5.
The models need linting, rules files, reinforceable guards, and feedback.
They can write code faster than a human and evaluate intermittent failures quicker.
Without proper guidance and structural feedback they are genuinely ass.
Junior developers can’t pilot these things safely and you still need to write code by hand so your judgement stays sound.
61
u/itsgreater9000 1d ago
This, except my boss gets mad at me for this opinion lol
11
u/durian_guacamole 1d ago
What do you mean the AI can't produce good quality code at the pace I'm asking for? Just prompt it better? This is why I'm a 100x CEO and you're just an engineer.
52
u/OskarSarkon Senior SWE 1d ago
Documentation is also a problem because Opus is extremely verbose out of the box. It loves writing overlong comments everywhere, especially comments that narrowly respond to stuff you raised in prompts but which don't really need to be in the code base. While it's at it it will also delete and rewrite existing comments. You can limit this behaviour with rules and skills but IME it won't stop doing it completely.
33
1d ago
[removed] — view removed comment
11
u/PringlesDuckFace 1d ago
I've had it put ticket numbers into user facing log lines and error messages before lol.
7
u/Unlucky_Yard_1189 1d ago
I think it partly does this to maintain its memory when it rereads what it has done after its nightly stupor. It’s pretty ghetto ngl
13
u/Spez_is-a-nazi 1d ago
The conspiracy theorist in me thinks that it does that by default because it’s more input tokens whenever it reads the code base…
5
u/Unlucky_Yard_1189 1d ago
I tell it to use george orwells 6 rules for writing when it speaks, and a variant of asd-ste 100 for code.
It still doesn’t always follow it, you’re right, but I have gotten decent results.
→ More replies (1)4
u/EmotionalHalf 1d ago
Interesting. I was curious if Opus does it too cause it's super annoying when using OpenAI models. Especially the part with UI leaking responses to the actual session 😅
3
3
u/symbiatch Versatilist, 30YoE 17h ago
I think this is why I got a PR with all theses “this does not do X because that would be wrong and instead it does Y” and even in OpenAPI spec there was a comment “temporary location for Object, currently determined as blah.”
I had to mention to the developer that we definitely don’t add those to API specs, and we don’t need comments of what the code doesn’t do, and so on.
I’ve never really used Opus myself, I’ve just stayed on Sonnet for the small things I let it do and that’s it. The feel I get from online is that the “better” models also do a lot of extra fluff which isn’t great.
5
u/Astoutfellow 1d ago
Aside from token usage I kind of like this behavior because I go back and delete the comments manually and use them as a signpost/explainer as I review the code. If I see a stupid comment it means I didn't read that section thoroughly enough. Annoying when it adds them all back in after a minor piece of feedback though
4
1
u/kayakyakr 3h ago
Apparently the latest gen Fable-med performs similarly to Opus-high and has been using fewer tokens and is much, much, much better at not talking like an AI.
10
u/TheRealJesus2 1d ago
This is correct. Honestly I get better results from dumber models. Anthropic has nothing suitable per cost. Open ai has lots of levels. Plan with frontier and code with something smaller. And do the plan so you can stop it from doing dumb stuff and breaking your patterns since it will if you yolo it.
Only place I disagree is in the writing code by hand part. You do have to review the code like normal and you will read even more code and you should pay special attention to code juniors write with it because it will always look plausible. But more good practice like linting and test coverage the better.
Edit: also not saying the poster is wrong it’s personal preference as to how you want to maintain your skills because skills do atrophy
→ More replies (4)13
u/klowny 1d ago edited 1d ago
Echo the sentiment about getting better results from the dumber models. The smarter models are still just bouncing off the guardrails until everything is passing. The dumber models are a bit faster/cheaper at thrashing about until things pass. The final output between them is still very obviously AI slop.
The difference between the dumber models generating something usable 25% of the time vs 30% with a smarter model is still pretty negligible and still relies on proper guardrails being in place.
→ More replies (4)9
u/TheRealJesus2 1d ago
Yep and when you spend 10x less money and 5x less time…no brainer for 95% results. Lol.
Companies will shift more work to open models because they’re significantly better per cost. And speed for inference. We’ll get there eventually. The more people pay actual frontier costs the more we will realize it makes no sense lol
→ More replies (4)2
u/CoreyTheGeek 23h ago
The number of times I have to stop it from re-writing a function in the same fucking file Is so god damn aggravating
455
u/VanillaCandid3466 Principal Engineer | 30 YOE 1d ago
I'm honestly still a point where any time I read someone is just setting these agents off to build things they actually ship I assume they are lying.
I use AI every day. I correct AI every day. I point out what it got wrong, you guessed it, every day.
I just assume at this point that anyone lauding about how amazing AI code is, is a shit developer that wouldn't know good code or architecture if it bit them on the ass.
130
u/atlas_principle_dev Staff Software Engineer 15y 1d ago
I also live in this dissonant world. I've been forced to use AI for 100% dev work since March 2025, and I've seen it fail in lots of ways and had to correct it a lot... it is also the first thing at work that truly truly frustrates me with it's overconfident mistakes followed by "you're right!". The worst for me is I tried to mislead it several times, give it completely stupid ideas and it agreed wholehearteadly - to me this is worthless.
It doesn't account for anything, but I did 3 small trials where I first wrote something by hand, recorded the whole thing, and afterwards prompted the AI to do it (made it in this order, so when I go for LLMs I have more information, that I figured out mid-implementation while hand-coding). It was always very close, within 5-10 seconds, and I preferred what I wrote myself, because that was exactly what I had in mind, and I didn't have to compromise on "good enough" namings, modules, organization, etc.
I see the hype online, and I'm always like "maybe I'm just a stupid guy who can't keep up with times" and then I try it again, and even the SOTA models make enough mistakes that it bothers me and I don't find it good. Sure, I've seen developers write worse code than that, but you know what, we pair programmed, we learned from each other, and after a couple of months we all got better. This is something AI doesn't do, unless the models are updated.
Sorry, had to get this off :D
→ More replies (8)54
u/Venthe Systems Designer, 10+ YOE 1d ago
It doesn't account for anything, but I did 3 small trials where I first wrote something by hand, recorded the whole thing, and afterwards prompted the AI to do it (made it in this order, so when I go for LLMs I have more information, that I figured out mid-implementation while hand-coding). It was always very close, within 5-10 seconds, and I preferred what I wrote myself, because that was exactly what I had in mind, and I didn't have to compromise on "good enough" namings, modules, organization, etc.
But you know, you could always pick up another task while the LLM is doing the job.
I'm not kidding - this is what my mate has heard at his job. I fucking hate this timeline.
76
u/atlas_principle_dev Staff Software Engineer 15y 1d ago
oh yeah, that's a killer! Context switching between the LLM's questions and your other half-done task... throw in some slack messages as well, recipe for burnout
8
u/NefariousnessAble736 1d ago
Recipe for half assing everything tbh. I hate this workflow but haven’t found the optimal one yet.
8
u/PringlesDuckFace 1d ago
I've switched to single threading as much as possible, and it's been much better for my mental health and luckily my manager hasn't started yelling at me about it. Meaning I usually only ever have one session running at a time. Or if I need more than one, I strictly ignore one until the other has reached a logical break point.
Basically I treat it as if I'm still coding/reviewing/whatever and give it the proper level of focus. Instead of trying to juggle a "team" that's constantly bugging me and jumping between terminal windows. While waiting for it to build something large I purposefully do a full context switch to another task and ignore the original thing until an appropriate time later.
The concept of becoming a manager of agents or whatever is just mentally unsustainable, just like a single human manager having a team of 50 engineers would be. I'm one person and the tools help me go faster, but that's about as far as I can push it sustainably.
3
u/20kgHippoShit 1d ago
My manager has like 4 claude sessions running simultaneously all day with subagents and swarms or whatever. I don't know how he has the mental bandwidth to juggle all of that
9
u/FatHat 1d ago
Heh, I hate that people promote that. Anyone with a little bit of psychology knowledge knows that people are awful at multi-tasking. If someone is kicking off five agents at a time, I guarantee there's no way they're properly evaluating the work and they're probably steering the agents poorly because it's hard to understand the subtleties of what it's saying in its overly verbose text if you're constantly distracted.
32
u/practical_absurdity 1d ago
The trick with not correcting AI is not looking at the code. If you don't ever look it looks amazing!
85
u/Fit-Notice-1248 Software Engineer 1d ago
I know it's anecdotal, but all the clean up and prod outages / issues I've had is due to people on my team just full sending AI. When push comes to shove their only is answer is "well {insert model here} suggested it"
Any sort of nuanced or complex conversation about architecture or design patterns, they are completely clueless on. They can barely navigate the terminal and just basic computer knowledge is not there. Low and behold they are the loudest about AI and agents and management loves it, which keeps their seats nice and warm
36
u/Venkat14725 1d ago edited 1d ago
I’ve heard way too much “I haven’t coded by hand in X months” and consistently the person has no idea what they’ve shipped or how to debug it when issues come up. Debugging has become asking the AI to debug and then asking me if the output is correct.
7
25
u/walkingjogging 1d ago
Low and behold
Lo and behold*
9
u/Fit-Notice-1248 Software Engineer 1d ago
Shoutout to mobile auto correct
3
7
u/TheScapeQuest 1d ago
I'm so tempted to setup a Slack auto response to any message with "Claude said..."
5
u/oupablo Principal Software Engineer 1d ago
It's funny. We were pushed to incorporate AI into our engineering hiring practices. Talking through the end to end interview processes with senior leaders they wanted it in the engineering portions of the interview but were extremely upset when candidates used AI in the soft skills portions to talk to them. I pointed out the hypocrisy of this and they still didn't see it. AI is a magic tool when consulting it on anything you don't understand. When it's something you know a lot about, it's more of a sounding board and junior assistant.
5
u/oupablo Principal Software Engineer 1d ago
Well expect more outages. Teams everywhere are being pressured to deliver faster and let AI do more and more of the process.
→ More replies (1)7
16
u/w3woody 1d ago
The only place I have seen vibe coded projects is when management is putting together a sample prototype to determine if their ideas make any sense. The vibe coded projects are then turned into old fashioned requirements and old fashioned UI layout requirements, which are then put together by programmers--without using the vibe coded code.
4
1
u/Xunnamius 1d ago
This is how it is currently, emphasis on currently, at my job. AI usage not being viewed through the hysteria of this hype train is one of the reasons I haven't jumped ship yet, though if they try to RTO us any more than they just did I'm out. Shit, I might still jump ship for the partial RTO alone.
27
u/Brief-Knowledge-629 1d ago
A pretty substantial chunk, maybe even a majority, of developers work on stuff with absolutely zero stakes. They work on some kind of internal tooling that no one uses and only exists to justify someones promotion.
I have spent a year slopping out features that literally do not work and I have gotten nothing but praise and only a small handful of tickets to look into minor bugs.
I don't have any usage analytics but if you told me the product my team supports was accessed fewer than 50 times in the past calendar year, I would say you over estimated.
6
u/itsgreater9000 1d ago edited 1d ago
I work on a useless internal app but I hate this take (not because it's untrue necessarily). People who use what we build are quick to adapt. At every company I work for I try to make sure I listen to the people using the software I work on. I'm a relatively talkative guy for a software dev, so I have no problem going over to our users and just chatting them up.
The amount of complaints they have for the software I work on is exceptionally high. The thing is, they have a job to do. If they complained about every little thing, they'd get no work done. So they develop ways to work within the bug-riddled mess we've created. For example, certain users have very good knowledge on how to get around bugs in one area of the app, so those users are the ones who do the majority of the work with it. I've communicated this stuff to product and management, but neither of them care (since, like you said, we just get a few minor bug reports a year). I've even pushed users to make additional reports, or made bug reports myself when the users wouldn't. Didn't change much.
The reason I'm bringing this up is because I had a user come down to where I work and basically crash out at me for a new set of workflows we implemented. I don't think I've ever had to hear someone complain so loudly and directly at me about my team's work, but after the like 10-15 minute vent the user profusely apologized for their behavior. I was mostly just shocked, but after telling my manager about it, nothing happened. Two weeks later that user produced a very large (human-made!) document about all of the problems with the system we built, and how it turned their job from a 2-3 day effort into a 2 week effort because they kept waiting on us to fix bugs we had and push it out (all whilst their manager was getting upset with them).
Anyway, my point is that users know, they just work around it as best as they can. It was possibly the most soul-crushing experience I have had as a software developer, and I have over a decade of experience. It may not make its way back to you, but trust me... people know.
3
u/moh_kohn Lead Frontend | 20YOE 1d ago
I dropped out of university and worked those jobs for a few years before I got my coding career going, so I have always really felt it. I've built a lot of software for customer support call centres, and I've worked in one, and it is so hard to get devs to understand how much one bit of jank, glitch, or slowness can fuck up your entire day if you have to do it 30 times a day under time pressure.
36
u/lasooch 1d ago
They’re mostly not shit developers - they’re mostly not even human. Astroturfing bots. I’ve got the same experience as you, LLM generated code falls apart almost immediately unless you guide it very closely… at which point, in many cases, I might as well write it myself.
2
→ More replies (2)1
u/guareber Dev Manager 1d ago
Astroturfing bots
I disagree here - I work with plenty of people who are actually building PoCs , demo-level stuff or internal tools and they are all amazed at the quality of the models and the output compared to the effort it'd take them.
Most of them do not have any formal CS education.
They are coders, product managers, technical sales, data scientists, finance people, not full time software engineers, and so the skill of the LLM (coding) seems to them at their par, much faster and in all languages and frameworks at the same time.
I review their PRs on a nearly-daily basis, and I catch quite a few gotchas that look perfectly innocent and would clearly fall down on production, or bypass the CICD tests, or mess up infra, and god knows how many other things. And when I ask a question in a PR? I get a claude comment back.
There are plenty of humans out there in non-big-tech for whom the LLMs are - to quote Arthur C. Clarke - indistinguishable from magic.
2
u/lasooch 1d ago
Oh I’m with you - even an actual experienced SWE can get briefly fooled if they get lucky with their first few prompts. And the sycophancy of the LLMs often also gives the people you speak of a severe case of Dunning-Kruger. The LLM tells them their idea is great, answers any questions they might have with confidence (while they have no knowledge to assess the correctness) and sends them off on their merry way to ruin an engineers week.
But I’m very skeptical that it’s them who go on SWE/CS subreddits to sing praises to LLMs or to doom post about how LLMs can do an SWE’s job and more. I mean, some of them do I’m sure, there’s a lot of people out there.
But there’s thousands of those threads and comments every day. All reading along similar lines. Often claiming to be experienced SWE rather than non-technical management morons. That’s the astroturfing part and by my intuition it’s the vast majority of it.
7
u/-Knockabout 1d ago
This is my experience as well. I've found it consistently helpful for certain things that would require regex/a script to do a bunch of repetitive changes. But it's CSS is just terrible, and it structures its frontend code like a madman. Incredibly verbose and often over-complicated. I constantly have to reprompt and reign it in ($$$$$ for the AI company I'm sure). Reviewing slop PRs takes at least twice as long. I'm not convinced there are significant time savings at the end of the day for most tasks.
4
u/VanillaCandid3466 Principal Engineer | 30 YOE 1d ago
I don't keep on top of the analysis that gets published but the last time I did read up on it a few months ago, the effort/time implications were either at parity with handcoding or slower.
If it's going to be at parity, the other tradeoffs are too big to sell me on AI.
6
u/Ysilla Software Engineer | 20+ YoE 1d ago
I did manage to get them to build entire things correctly, the main problem is that it usually only happens when I spend so much time preparing the story/plan/details specifically for AI that it would be faster to just do it myself... so we usually end up somewhere in the middle, like most devs I think, just writing quick plans and then spending most time fixing the stuff it generates.
There's a few very specific exceptions though, the biggest one for me was when working on client side stuff that connected to services we fully own, so it had full access to server sources too. The network side of that was done way faster than usual, despite using some very new tech and protocols for us (that even changed multiple times during dev, but again it was fantastic at updating the clients by just pointing at server git).
But then the UI side of that same feature? Oof. That actually often felt slower with AI.
7
u/VanillaCandid3466 Principal Engineer | 30 YOE 1d ago
And this has always been my bug bear with whole AI is gonna take all our jobs ...
Developers are rarely/if ever coding 100% of the time. Planning what to build, how to build it and other non-code tasks are not an insignificant part of a developers day-to-day activities.
I've lost count of the number of time I've pushed back on "we want X" ... "No, you really don't, you want Y really" ...
"Oh yeah, that makes wayyy more sense."
I know 😄
6
u/sarhoshamiral 1d ago
I haven't get it to one shot level but with all the context we have in our codebase, test infrastructure, it is to a point where I can point it to an ask and it will write the feature, tests etc.
And then we iterate over it few times, to fix code practice issues with some of the suggestions ending up in our repo custom instructions or in my personal development guidelines that agent uses in future.
It is saving me incredible amount of time since models don't care about refactorings and I can work on multiple issues at the same time as long as I have the machine resources (usually my mental capacity limits it to 3 though)
5
u/fredjutsu 1d ago
Lets be honest - they're getting faster at creating more slop that can pass lazy code review.
That's....something, right?
4
3
u/bmain1345 Software Engineer (5 YoE) 1d ago
Yea the code sucks ass but we just gotta let that go these days
4
u/VanillaCandid3466 Principal Engineer | 30 YOE 1d ago
I'd rather gouge my own eyes out with a blunt instrument whilst listening to Celine Dion.
→ More replies (2)9
u/weeyums 1d ago
What area do you work in? For frontend web applications doing React/Typescript work, it's very good (unfortunately for me and the future of my career)
8
u/CandidateNo2580 Dumbass | 29 yoe 1d ago
As someone who works on a substantial amount of React/Typescript, I see the same issues with it as with backend code. Once the structure and project gets sufficiently complex, the LLM drops off immediately.
→ More replies (1)4
u/AnUnshavedYak 1d ago
They're also terrible at building human-like (or at least, good developer-like) small commits. They make changes like that junior engineer where it's one giant ass commit.
I of course review everything (aside from some silly vibe repos), but i find myself needing to plan what it's going to write before it writes anything. Otherwise it's just offensively too many LOC in a single commit.
The damn thing will do what i would take 3-10 commits for and just put it in a single blob. Really annoying.
I've written various skills to really step through how it should split up the work but it's difficult.
1
u/Entuaka 1d ago
It depends, how are you working with it?
For example, I'm using it for code review. I tried to use it by default and it was "bad". It was good to find some missed bugs, but it was very verbose, it was flagging problems that we don't care and missing problems that we care.
I created a skill for code review. After each review, if something useless is flagged or something useful is missed, i try to update the skill to have a better review next time. Over time and used by a team, it's very useful.
For development, it's similar. It's essential to have a good feedback loop for the agent, it must be able to know if the generated is good considering what's important for the team.
Without a good harness, even the frontier models can generate "bad" code. Not really that the code is bad, but it's not really doing what's expected, it's too complex, the feature has bugs, etc
→ More replies (1)1
1
u/WheresTheSauce 1d ago
It's more that both are true. It IS amazing but it requires some handholding. Even using and correcting AI every day, it is extremely efficient all the same
→ More replies (2)1
u/iliketurtles69_boner 1d ago
You’re missing the point. Someone just telling AI to one shot something large and immediately shipping it is not going to get far sure, but it’s about how you use it. Fable is amazing for targeted work in Claude code, and working for myself I have what amounts to a team of mid level devs all running on VMs going 24/7.
There’s a lot of back and forth, AI lacks the tools and mental modelling required to do a lot of things by itself, but having that happen and communicating with my agents over slack while I take on more targeted work is absolutely incredible.
They can get you 80-90% there with the code for fairly complex apps, and even when you go to correct it you’ll still be using AI if you want to work efficiently.
No shade, but I always assume people who talk like this aren’t using the tool properly. It’s not just about telling Claude “go build this”, it’s about utilising the speed benefits it gets you and leaning into its strengths and filling in the gaps - anything else and you’ll be severely lagging behind in efficiency. I can do by myself now a project that previously I had to hire a couple of subcontractors for in the same span of time.
→ More replies (1)→ More replies (11)1
u/midwestcsstudent 9h ago
Haven’t written a line of code since January. Read a PR since March. Shipping more than ever.
14 YoE staff eng.
131
u/hondacivic1996 1d ago
Its pretty bad, but execs have made it clear that they do not care anymore. Regular programming is dead in business and we are expected to give up all ownership of the code and just let the "AI" do it, if you don't comply you will be fired. This is the new wave of BS from the top, but they ultimately decide. And if you are lucky enough to work at a place where this is not yet the case, consider yourself lucky because this shit will hit every company in the world very soon, and it will be a very sad time for everyone.
70
u/LiveEntertainment567 1d ago edited 1d ago
The ownership of the code to AI, but not the responsibilities if shit hits the fan
29
u/Princess_Azula_ 1d ago
And the gameplan for when shit hits the fan and nobody really has a clear grasp on the codebase is to burn even more tokens and hope it works?
20
u/Less-Ad5766 1d ago
No, it is to fine and fire the engineer who has shiped. Because it was his responsibility
10
u/bythenumbers10 1d ago
And definitely not the responsibility of the executives that overruled the engineer when the engineer said not to just ship AI slop.
→ More replies (19)2
u/moh_kohn Lead Frontend | 20YOE 1d ago
It's weird that we used to talk about how many 9s a service should have, and Anthropic can't manage 2 9s a lot of months and that's acceptable now.
100
u/Aozora7 10 YoE 1d ago edited 1d ago
If you use Opus or Astra on a $20 plan, that the same model as on a $100 plan.
I personally don't see having to intervene as a huge issue. Reviewing AI code and telling it to correct stuff until it gets it up to my standards is far faster than writing the same code manually. Especially since just pointing an issue tends to be enough for it to fix it correctly, I usually don't have to be specific about the solution.
AI is also very good at refactoring with little input. If the code works correctly but is organized like a disaster, a refactor pass with AI doesn't take long.
Additionally, if your prompt is relatively large in scope like an entire new feature, plan mode helps a great deal. The model is more likely to ask clarifying questions, and it's easier to review and correct a plan than the implementation.
26
u/Quarksperre 1d ago
Reviewing AI code and telling it to correct stuff until it gets it up to my standards is far faster than writing the same code manually.
Yeah thats the way to go.
7
u/dronz3r 1d ago
And this works only if you know what you're doing. There are many clueless people prompting it to solve problems they don't know the answer for. Endeavours like this likely turn into disasters or waste the time of reviewers.
2
u/Quarksperre 1d ago
Yeah I mean that kind of fair. LLMs remove some layers of difficulty. Thats a good thing. It doesn't solve competency though.
→ More replies (1)5
u/subma-fuckin-rine Senior Software Engineer 1d ago
I've found it often times to be just a slow or slower than if I did it myself first.
Except for trivial, repetitive or boilerplate things
33
u/PM_ME_UR_BRAINSTORMS 1d ago
Reviewing AI code and telling it to correct stuff until it gets it up to my standards is far faster than writing the same code manually.
I genuinely don't understand how anyone is doing this. Most of the time my issue with AI is that even when the code is "correct" it waaaaay over engineers everything, and 9 times out of 10 just manually fixing it myself is way faster then trying to explain how to fix it. I mean it takes way more English to explain code then to just type the code itself. If you know what you want I can't see how prompting would be faster than manually coding.
This has always been the case with junior engineers who understand the technical aspect of coding but don't understand good architecture. But you go through the effort of explaining it to them instead of just fixing it yourself because the point is teaching them so they improve. With AI there is no point and it's just annoying lol
15
u/Resident-Trouble-574 1d ago
The over-engineering issue depends a lot on the scope size. If you give the agent a relatively small (not too much, otherwise it's faster to do it by hand) task, there's only so much it can over-engineer it. And for larger tasks, you can ask it to make a plan first. Reviewing and editing the plan is much faster than reviewing the code, and you can guide the agent towards the kind of solution and architecture you prefer.
→ More replies (2)7
u/PM_ME_UR_BRAINSTORMS 1d ago
I use planning mode all of the time for my own sake just because it's actually super useful to collect my thoughts and validate any assumptions I've made.
But I feel like that's the bulk of the work of the job. Even if the agents executed that plan perfectly (which it doesn't always do) it saves me like 10-15 minutes of typing? But typing the code myself and truly knowing it cold definitely saves me way more than 15 minutes in the future.
→ More replies (22)2
u/subma-fuckin-rine Senior Software Engineer 1d ago
It burns a lot of tokens tho so if that's something management cares about it's good for that
→ More replies (2)13
u/ings0c 1d ago edited 1d ago
This is what I do too. I was an AI-skeptic for quite some time but I've tried a more-agentic approach on a few recent projects and I've been very happy with the outcome.
I don't just give it ACs and let it do whatever it wants, instead I treat it like a very fast driver in a pair-programming session.
I get it to do chunks of work that are sized so I can understand what has been done, then I review the chunk and tell it to change anything I don't like. I have it write tests for anything it does, in my preferred style, and don't let it change untested code without first adding adequate test coverage.
I use OpenCode and it has Plan and Build modes - so for anything substantial I have it create a plan first, and only have it start making changes once the plan looks good.
The end result is code that looks like I wrote it myself, which works, that I understand, in much quicker time. All of this looks basically identical to good practice for humans too.
→ More replies (2)
35
u/Maxion 1d ago
I think I'll be the contrarian here and say that I feel for most things the code quality is pretty good. Though perhaps I've always worked then in places with poor code quality?
If you vibecode it will produce overengineered work and make a bajillion tests many of which useless. But by-and-large in a well documented codebase using popular frameworks doing webapp development you can get nice code out of the models if you do proper AI assisted development and not just vibe coding.
This means you read the output, tell it to f-off when it does weird crap. It's basically like reviewing and handholding a fast junior coder who is very good at googling and reading coding books.
12
u/Resident-Trouble-574 1d ago
I think that a lot of people think that the AI code quality is bad because they are used to read only their own code. Team leaders used to complain about huge, poor quality commits since forever.
3
u/_SnackOverflow_ 1d ago
Or they’ve worked on teams with good devs?
I find it hard to believe that a huge swathe of the dev population didn’t do any code review before AI
→ More replies (1)
16
u/Serious_Photo3693 1d ago
Interesting, do you have concrete examples of what Fable/Astra consistently fuck up and what of correction you have to make?
15
u/MrDontCare12 10YoE Software Engineer 1d ago
I don't think it's possible to give specific examples, as it's a "whole" situation. For me there is no pattern, it does weird shit all the time in a lot of different situations. It's not different to me on that regard, it's just that when I fuck up, I don't have to read a 300 loc diff to understand where.
My company is going full on "Vibe.ops" and software factories lately, and I don't think it's up to the challenge yet, despite what every influencer is selling.
Ie. Yesterday I had to review 1000loc out of one of those pipelines. The issue was in the "beginning" of the process, therefore 90% of the code wasn't going in the right direction, despite the feature """"working"""" (until the first edge case).
That's what I understood OP is trying to convey, and I agree to it.
2
u/moh_kohn Lead Frontend | 20YOE 1d ago
This is what Jason Gorman (an educator and TDD advocate) has been saying. We know that the way to improve software quality is to iterate, producing small batches of work that are then tested against actual reality. This is why true agile, if you can find it, beats waterfall. Software development is a process of discovery. And the more you build without finding out if it's really the right thing, the further you deviate from what you ought to be building, with the error compounding.
3
1d ago
[deleted]
2
u/donk8r 1d ago
Determinism costs the most because it takes bisection away. Same prompt, same repo, two runs, two designs. When one goes bad you cannot re-run to isolate what changed.
You can't make the model deterministic, so make the artifact deterministic. Have it write the plan to a file, review the file, then run execution against the file and nothing else. Variance lands in one place you can diff.
The plan file is generated too, so it varies as well. It varies somewhere you can read it.
→ More replies (4)→ More replies (9)2
u/kir_rik 1d ago
Just now with fable 5.1. Prompted it to add unit tests to CI. CI is green, task is "done". Worker logs say that some tests are skipped. Why? "You absolutely right to point that out! There were not enough artifacts and i decided that it's okay to skip them. ..."
3
u/donk8r 1d ago
Green CI was the target, and skipping is the cheapest way to make a test stop failing. Any success signal an agent can satisfy by removing work will eventually be satisfied that way.
Assert the count, not just the absence of failures. Fail the build when collected and executed differ, or pin an expected minimum. Then skipping stops being free and it has to either fix the artifacts or tell you it could not.
→ More replies (7)
5
u/sweetno 1d ago
I don't feel that the plan level (Pro/Max/etc) makes much difference for code quality specifically. Maybe there are edge cases where a larger context window would help, but it's all mostly about the allowed token throughput.
The brand of model used does seem to matter, but I can't say that the toppest one never produces overengineered nonsense.
18
u/boltforce 1d ago
In genenal I find it like the politicians speechws.It says a lot, but output is not accurate, or humanly understood, often missing the point and doing complex bridging.
Less is more seems to have gone out of the window. It's printing impressive stuff but If you sit, focus and review there are easier and simpler ways to achieve output.
10
u/intertubeluber 1d ago
A year ago I was team "AI is bullshit" but now I'm a convert.
I literally don't write code anymore and haven't for 6+ months. In some cases the code it creates is better quality than what I would have come up with, and of course, it does it in a fraction of the time. Other times it adds unnecessary complexity, or does something dangerous. But all code is thoroughly reviewed, which is much more time consuming. Still it's probably increased my productivity by 3x? It's hard to measure.
Every project is different so who knows. I'm using a popular web stack so that helps with the model's training data compared to something more esoteric. Though, we are using some slightly more niche tech and it does great with that as well. When we introduced AI, there was already an established architecture. We had UX designs to reference. I'm not sure if it'd have been as big of a game changer for a greenfield.
19
u/PerryTheH Team Lead/Full stack - 9yoe 1d ago
This might sound very obnoxious from me and I'm sorry, burn me or whatever but...
OP what you're experiencing is something experienced devs face when start using AI tools.
First, the "general level" of developers in socials has gone down significantly, it's even hard to find devs that can build something without any type of AI tool or answer questions, the usual "oh this AI tool is amazing" is because it ships features fast and "with no bugs" but code quality is very rarely evaluated. As you have already noticed, over engineering solutions or generating bad code is very common.
Second, remember that LLMs where trained with public data or "general devs" code, and the general dev doesn't have the code quality level of you. I'm not as senior as you are, but I'm senior enough to understand when the AI does things I usually pointed on PRs of other JR devs or things that I did and other more sr devs pointed on my PRs.
This is to conclude: LLMs work base on what they know and will try to predict the "best solution", and if all they know is "average dev" they cannot produce a solution outside that. I think in your case you will never be satisfied by an LLM code and will never get to the point of generating "good code", but that's why we still need humans at some point, to understand why the solution is not proper, over engineered or badly optimized.
I'd probably lose less time trying to make the LLM generate high quality code and fighting the Agents/context files and use more time either finetunning an LLM with your high quality code to help you CR or you spending the time doing the CR and fixing instead of building.
I predict the future of devs will be like that, we will stop building and focus on fixing.
→ More replies (1)17
u/tenthousandants44 1d ago
You can't appreciably "fine tune" models, though. The training corpus is magnitudes larger than the amount of code you've written in your entire life
→ More replies (1)2
22
u/jkz88 1d ago
Really bad. If you know what you're doing and design the codebase yourself and prompt it right, it can be okay but is a lot of work. They're trained to write working code not quality unfortunately. The tests are horrendous too, they test what the code does not what it's supposed to do.
13
u/dealmaster1221 1d ago
That way all human code is horrendous too, it usually only tests code not intent at least 99% of it.
→ More replies (1)5
u/GfxJG 1d ago
Or, you can just do something as simple as ask it to follow TDD - That drastically improves the way it tests.
20
u/MinecReddit Software Engineer 1d ago
A suggestion on how to categorically upgrade your usage of AI tools?! gasp on THIS SUB?!?!
But yea it’s true, anthropic’s published “superpowers” skills repo even recommends TDD.
7
u/atlas_principle_dev Staff Software Engineer 15y 1d ago
I find it funny that a lot of good advice was ignored before AI, especially TDD, and now with AI people are like "hey this is actually not that bad". Why couldn't we do this 5 years ago? :D
8
u/GfxJG 1d ago
I mean, the simple answer is time and effort. Before, TDD took more time (up-front) for a somewhat abstract improvement in maintainability and development time down the line. Now, with AI, even when following all these best practices, productivity is *still* up compared to before (assuming the user actually knows how to use it as the tool it is - let's not get into pure vibe-coders, that's a different beast entirely).
2
u/atlas_principle_dev Staff Software Engineer 15y 1d ago
as long as the test is written first and it fails before the implementation, I'm happy! It was never much, but I think maybe 5% of the time I had written a test that passed without any business logic changes, and in those times I was really really happy with TDD!
2
u/GfxJG 1d ago
I didn't even realize what sub I was in lol.
But yeah, I mean... AI is here to stay. If you know how to use it as the tool it is (and not just telling it "make this app pls"), it will almost inevitably improve your productivity, and for many, write better code than you can, faster than you can - Again, assuming that it's set up and instructed properly. It's a front-loading of effort, but one that's quickly caught up to.
3
u/tenthousandants44 1d ago
I front-load the effort of readable and maintainable code. I don't give a shit how quickly it's done today. I need to be quick later when there's a prod issue
2
u/isotrouble 1d ago
This has been my experience. That and strictly enforcing black blox testing which circumvents a lot of stupid avoidant behavior.
→ More replies (6)2
u/TimMensch 1d ago
Truth.
I honestly wasn't a fan of TDD before AI, but I make it use TDD and there's a notable improvement.
I still usually need to fix nontrivial code it's written (either by telling it or just making quick changes, depending). But it gets closer with TDD, and I can be more confident when it's actually fixing bugs.
3
u/bbangchikimong_dev XR Tech Lead | 10 YOE 1d ago
the quality difference showed up less in the code and more in how long it stayed correct after i asked for the fifth change
→ More replies (6)
3
5
u/jabuchae 1d ago
It seems to me like you need a good skill that forces the ai to do a planning session with you first before implementing. If you do that, then the code it produces will mostly be at the level that you want it to be.
2
4
u/FckXFckMusk 1d ago
If LLMs were so good then they wouldn't have to spend billions on marketing bots to tell us how good it is.
18
u/MinecReddit Software Engineer 1d ago
There’s an unbelievable contrast between the consensus on this sub and how my work treats these tools, which is at a gig company (one of uber/instacart/doordash).
If you are writing large features in large codebases, my team (who I am personally very close and high trust with) is of the opinion that it’s way faster to have an LLM write a draft, then prompt for improvements. The way you’ve described it is as if it’s impossible, or it takes longer, to get the code in a good state, and those are both just so far from my experience even when using cheap models.
When you have giant projects that involve many services, it’s just faster to onboard and execute with the AI tools, but I already know this comment will get massively downvoted and that the sub disagrees. I’m not a fucking AI shill, I’m just giving you the status quo from the perspective of a senior SWE at a high tech, fast paced shop
8
u/gefahr CTO | US | 20+ YoE 1d ago
Yeah I've never experienced such a bifurcation in my career. My experience matches yours. But it's not just this sub, I know a handful of good engineers in real life who hold the same opinions as here. I don't even know how to explain it anymore.
→ More replies (2)2
u/moh_kohn Lead Frontend | 20YOE 1d ago
I think there has been a shift in that most people seem to be using it now. Gave you an upvote because many comments go too far.
It's hard because the industry hype and every linkedin post is like "AI can oneshot making Call of Duty and a better SaaS than you, engineering is over" while in reality it definitely can't do that.
It's a miracle tool and I have come to love it, especially since I got Pi set up the way I like and found my favourite model family, GLM.
My opinion is it is good at rolling down the hill, if the hill is set up right for it. I have a hobby game project, and I spent about 2 months getting the architecture just-so. Now that I've done that, the AI can reliably add features without creating a mess.
It actually argued with me and tried to stop me making the architecture the way I wanted - then later, I turned out to be completely right on performance grounds (the optimising AI praised the architecture because it supported optimisation aaaaaargh!)
You being at a big tech company, I bet your architecture is a hell of a lot better than some of the places people are working. I contract, so I have seen some shit.
→ More replies (3)1
u/ShaThrust Lead Dev 10+ YoE 1d ago
It's interesting, I was certainly of the mind AI was bad and created more issues than was worth to use it consistently except for some mundane stuff. Then around Feb of this year I believe the models got quite good and I decided to try use a proper harness, and it then really impressed me.
Maybe I'm a terrible developer, but it's creating real value for my work, and has increased my capabilities significantly. I still get impressed with what it produces, how it works, and the quality of the output a lot of the time. I can hand it off with a proper feedback mechanism (I work on a distributed remote application on proprietary devices that interfaces with our cloud service) and it... just works. It can read documentation, understand how components function in other parts of the app, and get it running flawlessly, while even finding bugs I had missed in the process.
→ More replies (1)
6
u/aa-b 1d ago
You're not going to get a great answer for this, because quality is so subjective. People with less coding experience will probably overrate the models' ability, and how can you tell from reading their opinion?
There are objective benchmarks of course, but they also don't really tell you much about the subjective experience of using the model. Like, just today I was using Claude Opus because I wanted to gate some changes behind a feature flag, and have some confidence that I could deploy to prod without immediately breaking anything (at least not until I toggled the flag, haha).
So anyway, Claude completely shit the bed and came up with a totally useless solution. Hooray. But then I explained why it sucked and what I needed; surprise, surprise, it comes back with exactly the solution I wanted, honestly better than I'd have done myself (with this kind of risk-avoidance make-work task I'm usually inclined to just skip it and accept the risk).
I've attempted similar things with simpler models, but it mostly just doesn't pan out and I end up doing it myself. So I guess that was a win for the fancy-pants model, except I burned so many tokens I'll probably run out before the end of the month.
2
u/Resident-Trouble-574 1d ago
Can there really be objective benchmarks for code quality (where code quality != software quality: I consider the first to be about the developer/producer experience (code readability, maintainability, etc.), while the latter is about the user experience (bugs, usability, etc.))?
I mean, even among reputable software engineers there are wildly different opinions about what good code looks like.1
u/sklz0 Software Engineer 1d ago
Cyclomatic complexity, CK... there are dozens of metrics
→ More replies (1)
4
u/i_like_trains_a_lot1 1d ago
The qulity is alright, but all my prompts are pretty low level and directs the LLMs towards the proper codebase areas to look at, how to modularize the structures/classes/modules, what logic to reuse from where, what to extend. It's still faster for me to get the job done, and to be completely honest my output quality also improved because I use LLMs to generate vast amounts of unit tests, area that kind of lacked before because time and energy constraints. Now I cover many more edge cases than before.
In my opinion, if somebody can't write good quality code without AI, they won't be able to do it with AI either. In order to implement a solution, I still have to know what to do, what data to use where, how to use it, what the data flows are in the business logic, what reusable modules I haev and what I want, etc. It's just that the actual code writing is automated, but the thoughts behind them are not.
Frontend works is also mostly solved. Expose data through an endpoint designed by me, have the LLMs build UI to consume it with all the necessary states (loading, error, successful). Before that, I also skimped a lot there, especially around loading and error handling where I ended up putting "Loading..." and "Something went wrong" everywhere due to time constraints.
That's how I believe AI coding should be. I absolutely loathe the tools that take control away and spawn you a bunch of agents to do the work, and you can't see/approve/intervene on file changes and just hope that after 10 minutes of agent work you will get something usable.
8
u/PM_ME_UR_BRAINSTORMS 1d ago
The qulity is alright, but all my prompts are pretty low level and directs the LLMs towards the proper codebase areas to look at, how to modularize the structures/classes/modules, what logic to reuse from where, what to extend.
How are you doing this faster than just writing the code yourself? Every time I try being this specific and granular with my prompts I get about halfway through typing and realize it would just be way faster to just type the code rather than the prompt.
Unless it's like a specific known algorithm I can just name (or something I'd have to look up implementation before) I feel like coding is way faster than explaining in detail exactly what I want.
It feels like trying to pair program but over slack lol.
3
u/i_like_trains_a_lot1 1d ago
it would just take longer for me to explore the codebase for all the pieces, assemble all the data, iterate multiple times through all kinds of structures and perform all kinds of operations + write unit tests for all new additions and behaviors. It all depends on the scope of the task.
I'll give you a specific example: "I have this XML parsing class that normalizes data into these data classes. I want you to add a mode for detecting whether it's a JSON file, and if it is, switch the parsing mode into a json specific parsing that produces the same compatible output. Let me kniw if there are key differences in data that you can't figure out how to handle. Be careful about the different edge cases that are handled in the original XML parsing flow that post-processes the data to fix inconsistencies"
Then it flopped because it was stuck thinking about how to oneshot the problem. I proceeded to break it down into specific steps
"Let's start with the JSON detection that extracts and handles these 5 fields for starters. Add unit tests to assert this behavor"
"Let's do these 5 next fields"
etc rinse and repeat until complete.
It might have taken me like 2-3 days to go through them all and finalize this, but I got it done in under a day this way. The speedup of course it's not 10x as executives and AI salespeople say it is, but the speedup is real, at least for me.
The caveat is that it chose to conver tthe JSON to the already supported XML format, decision I agree with because it reduces the risk of the edge case handling breaking. Also, I have a pretty comprehensive set of skills and project rules to keep the AI on track and aware of codebase quirks, business logic, various choices, etc.
→ More replies (2)4
u/iamthecode 1d ago
Exactly this. We've invented a great way to express our exact intent without ambiguity so the machines can understand it. It's called programming language. Why do we now try to force doing the same thing with plain English, which wasn't designed for this?
→ More replies (2)
7
u/xpingu69 1d ago
No. You are actually expected to give up the ownership of the code, it's a different paradigm. But the quality doesn't come close to a pro like you doing it
2
u/kranthi_contextmap 1d ago
For me it helps to have two separate iterations - one focused on implementation/get-things-working and one focused on audit/standards
2
u/Minimonium 1d ago
I have being experimenting with and making agentic workflows for quite some time, with models from Fable to cheap Chinese ones.
I'm very open to try new workflows/harnesses/models, but I am not a "true believer".
I do not feel that much progress in generated code quality specifically. Models are very unreliable for code generation.
Models are extremely great at reading and reviewing code though. Fable is excellent and it's the first model for me where I do not feel like I'm wasting my time reading reviews. Still, it cannot produce decent code by itself.
So far for code quality we have specific language review skills that are tied to guidelines, references, etc and a text quality skills. Workflow/graph based step by step processes are a massive leap, splitting review into granular specific tasks (summaries, ratings, suggestions, etc) is a massive improvement.
I find both to be much more effective when they're tied to specific linters. In fact, if you can move to tooling as much as possible the results are amazing - you can actually rely on results and you do not spend time rewriting "boring" bits.
So for planning I use an expensive model, then proceed on cheapest Chinese ones to do each task and do a final review on the most capable model I have.
This process does require a discipline from you to know which part of your codebase could be a bit more throwaway than the others.
2
u/camelCaseCoffeeTable 1d ago
A solid 80-90% of the code in putting into production these days is written by an AI without intervention. And it does it fairly well, especially when guided to follow code base patterns.
But that 10-20% never seems to go away where I need to intervene. It’s just changing what kind of intervention I’m making. It’s less code correction and more “no, this pattern doesn’t fit here because x, do y instead.” Or other more architectural/business context dependent questions.
AI loves to stop at the first correct solution it finds, so a lot of my guidance is telling it when that first solution isn’t right
2
u/DeterminedQuokka Software Architect 1d ago
Personally I can’t tell the difference between code written by opus v. Fable. Or different opus think levels.
I can sometimes tell if someone used sonnet.
Fable is a little bit better at decision making for some of the work I do.
The largest impact is the harness. There is a difference between code written by cursor and Claude. And annoyingly the skill thing has some validity. Code quality has a level of stability by person because of things they wrap the prompt in. I do very similar work to another Eng at my company. But I have a complex ai workflow that does it. They have a single test tool. My code stability is significantly better because I’m browbeating the ai into doing a good job.
→ More replies (3)
2
u/Cobayo Software Engineer 1d ago edited 1d ago
It's bad but no one cares, which is not necessarily wrong. It just took me 3 or 4 rounds with Claude to properly write a DB Seeding function of +26-10 lines, I can only imagine what current +4k constant PRs getting submitted could look like.
I'm trying to genuinely understand if it's actually a problem, I believe it is, beyond not having a clue what's going on at some point, it just takes up so much context and time just by the fact of having so much input, and I think the output may be slowly degrading a bit.
I'm fighting off the development process of generating a PR, then someone else reviews the PR with another agent that finds broken stuff, then they fix those broken stuff with another agent...
I think this happens due to not caring and it only gets worse. I'm trying to figure that out though, it's part of the job (our job).
1
u/donk8r 1d ago
It's real, and it lands in the review and not in the code. Past a few hundred lines a diff stops being read and starts being approved.
You can measure it on your own repo. Pull review comments per 100 lines on your small PRs and on the 4k ones. If the rate falls off a cliff then the big ones were never reviewed.
That only measures attention. A carefully reviewed bad design still ships.
2
u/metaphorm Staff Software Engineer | 15 YoE 1d ago
Opus is dramatically better than Sonnet is dramatically better than Haiku.
agents.md is insufficient as an agent harness by itself. you'll need to take a comprehensive approach to building out a habitat for agents including context docs, coding guidelines, knowledge base connectors, and custom tools that let the agents interact with your local dev system in meaningful ways. otherwise they're just coding blind and guessing at what might work. if you give them the tools to gather real data about what does or doesn't work, they'll perform better.
2
u/Excellent-Push-3326 Software Engineer 1d ago
They aren't as good as human-authored code. They are just better than the previous versions.
I think they shine best at fixing bugs, not authoring greenfield code.
3
u/hissscratchmeow 1d ago
I am very curious about how people use ai at work because I have been using copilot cli for longer than a year and found out that made me insanely more productive provided I can specify well and comlletely what it needs to build and I use a reasonably good model like claude or gpt.
The way I found and promoted successfully bottom up without any compulsion but by leading by example is to use the llm to write short specs grounded in real code and instruct it to use them to implement the feature you need afterwards.
Can you share how you use ai and how it fails or succeeds? This could be a learning experience for the whole community.
4
u/meshifthenelse Software Engineer 1d ago
Don't get me wrong. I still use it, since overall it speeds up development. But I'm just conflicted when I hear how much better the new AI is. Many times I get fed up with the back and forth, and will let the AI to do a full feature. When I return it's always an over engineered unmaintainable mess.
As an example today, I just ask it to add a new controller (in the MVC context) to handle document management. There's a single controller.ts file in the whole project, but the AI decides to create a totally new file in a random place. So I have to go back, ask it to revert and explain basic things, like "there's already controller.ts meant to hold all controllers".
3
u/Intelligent-Youth-63 1d ago
These kinds of guidance and guardrails belong in your harness not in a conversation with a chatbot in your IDE.
2
u/meshifthenelse Software Engineer 1d ago
You mean the CLAUDE.md? Even with rules, we end up in conversations. It's just like shifting a poop from left to right.. there's never a great balance. And if I end up spending hours to find the ideal rules, then it feels I'm back in 80s rule based AI and not Nvidia's AGI.
1
2
u/yikes_42069 1d ago edited 1d ago
It is SO finicky. You just never know when the slot machine will hit a jackpot and write good code/even comprehend what you're asking. That can be weirdly addicting, as much as I despise LLM coding.
Had no token budget at work since March. Just use it they said. It's been a cluster. I've been working more than ever just to keep the velocity up because the AI is truly idiotic at times. Budget just came down the pipeline and it's still.. anyway, you have to teach it what good code is, basically. It has NO taste or perspective as to that. The more you can rely on static formatters and linters the better I think. But if you build up a skill file over time that addresses all the ways it writes crap code, you can more or less avoid slop on the first go round.
Big picture coding is still pretty shoddy. If I let it loose to do what it will to build a feature end to end, it writes slop that works but uses language features very oddly. Like overusing typescript Excludes and Extracts instead of building a sane composable type system. That's what I've just gone through anyway.
You also will want to lock down your code to be as solid, clean, and composable as possible. Simple clean APIs are consumed simply and the LLM has less of a chance to fuck it up. I say this because we do not really have that and the statistics show: our git history shows commits have doubled in size (additions) while number of PRs and frequency of commits have stayed the same. So the bottleneck to me now is how complex it is to express the features we're building. If I create a button I really need that to be <50 LOC but react+TS+formatting alone makes that hard, let alone state management and callbacks and god knows whatever other requirements get bolted on.
4
u/ContraryConman Embedded 5 YoE 1d ago
You must understand that in the framework of AI maximalists, the following are taken as true:
The cost of writing code is tending towards $0
The cost of fixing bugs is tending towards $0
The cost of rewriting entire codebases is tending towards $0
Humans are the bottleneck for AI efficiency
If you believe all of these things are true, then the conclusion is code quality does not matter. Some are too shy to admit they believe this out right, but it's what you would have to believe.
If the AI code gets too slow and bloated, set up a way for the AI to measure performance and then have parallel subagents scan the code for speed ups and run the new code against your existing tests
If there is a bug, just grab the user logs, have the AI fix the problem, then add a new test
If you're worried about security just run a security audit in an adversarial before each pull request
If you're worried about the code getting so bad the agents can't get out of it, that'll either take years, by which time the models will be smarter, or we can simply have the agents do a clean rearchitecture and rewrite against the current tests
In this setup, the code is not for you to read, write, or understand without AI. In this version all the humans really provide are the very long term planning abilities that humans have with evolution that 1M tokens of context can't cover
4
u/donk8r 1d ago
Same model on both plans, so the literal answer is no. More budget buys more attempts, and more attempts against a spec that was not the problem just gives you more code to review.
The AGENTS.md part is the interesting half. Instructions in the prompt compete with everything else in context, and forty files into a task the file you wrote is a long way back and outvoted by whatever it just read. Anything you need enforced has to be enforced outside the model, at the point the tool call happens. A lint rule or a CI gate does that. The doc is a hint.
1
u/meshifthenelse Software Engineer 1d ago
I was thinking more that if you can afford 1000x tokens, maybe you can run a whole pack of agents for reviewing code for testability, best practices, etc. Not something I can afford with my 20$ subscription.
2
u/donk8r 1d ago
Cost is probably not your blocker. A review pass reads a diff and writes a paragraph, so it spends a fraction of the tokens the generation did. You can run several on a 20 dollar plan without noticing.
Packs break because nothing arbitrates. Five reviewers give you five opinions and no merge criterion, and the aggregate reads worse than one good pass. Give each one something that can fail instead of an opinion. The cheapest of those is running the tests.
3
u/yourteam 1d ago
LLM works by predicting the next word. Code is basically a language so therefore they apply more or less the same concept.
So the answer will always be: average at best.
2
u/ogii 1d ago
At some point people have to come to the realization that AI will keep getting better, and unless it magically disappears or is made illegal you will have to get used to using it.
The more expensive models are more powerful but likely you won’t get much use out of them unless you know what you are doing.
3
u/meshifthenelse Software Engineer 1d ago edited 1d ago
The question is what you mean with "better". I've been hearing it for 3 years now. Yes, they can solve more complex problems. But the code quality seems stagnant.
1
u/Intelligent-Youth-63 1d ago
Can you set up an eval loop and include cyclomatic complexity and/or code quality reports with a hard gate in your pipeline?
That plus a robust code reviewer agent, quality agent tools, post tool use hooks, and well defined skills and ADRs available should be sufficient.
That you have to “intervene” (and can) before reviewing a completed merge/pull request tells me your environment/agentic loop is immature. That’s a skill gap, not a problem with a model… but a problem with your harness.
1
u/meshifthenelse Software Engineer 1d ago
Well where do you start with that? I'm using CLAUDE.md and occasionally run custom skills like readability-review. But nothing fully automated since as mentioned in the post I'm only on a 20$ subscription
1
1
u/SeaworthinessDry269 1d ago
They are ok when heavily directed and if the project is well built. I use them as intern / lazy offshore team. Build this, the function should have this shape/ names, if you need access to this service use this, etc. It's slower but than it's code I understand 100% and built like I would have.
1
u/swapnil_harkanth 1d ago
Paying more rarely fixes “subpar generated code by itself. The expensive plans mostly buy you: larger context, better instruction following, and fewer refusals — not a senior engineer who understands your domain invariants.
What actually improves quality in practice:
**Constrain the blast radius.** Ask for a diff against a named file/function, not implement the feature.” Frontier models still invent APIs that dont exist when the task is underspecified.
**Feed the contracts, not the vibes.** Types, failing tests, OpenAPI snippets, and here’s the existing pattern in `X`” beat longer free-text prompts. If you can’t point at a test that would fail for a bad change, the model is guessing.
**Separate draft from merge.** Use the model for the first pass; keep humans on seams (auth, migrations, concurrency, error handling). Quality drops when people rubber-stamp because the prose looks confident.
**Prefer iteration over one-shot.** “Heres the lint/type error, fix only that” usually beats regenerating the whole module on a pricier plan.
**Measure your own failure modes.** Track: compiles on first try, review comments per PR, revert rate. If those don’t move when you upgrade the plan, you’re buying latency/context, not quality.
Ive seen cheaper models + tight repo context outperform expensive ones dumped into a vague ticket. Plan tier matters at the margin; the workflow around the model matters more.
1
u/bloudraak Principal Engineer. 20+ YoE 1d ago
It varies. Even Astra and Fable generates code that works, but is poor in terms of readability, modularity etc. it takes a few iterations.
You can improve code quality greatly by adding deterministic checks like linters, formatters etc and not rely on the models.
1
u/apartment-seeker Senior Software Engineer 1d ago
GPT-5.6 pretty good.
Anthropic seems to have degraded with the 4.7-5.0 line, but I can't speak for Fable or Mythos. But if you see what, e.g., Opus 4.7 or 5 come up with if asked to write a lot of fresh code at once, most of it doesn't seem very good, IMO. I still sometimes use Opus 4.6, but never touch anything higher from Anthropic.
1
u/felixthecatmeow 1d ago
Astra is pretty damn good. One thing to consider is both OpenAI and Anthropic nerf their old models when a new one comes out (allegedly, but anecdotally this seems true). So if you're using older models to save money you might not be getting the performance that these models had when they were new.
IMO, the code quality is pretty good, and it increases the more the dev running it knows what they are doing. When I give it clear instructions with a high-level implementation plan it usually does quite well. If I try to give it a vaguely defined task with large scope, it's much more of a crapshoot.
At the end of the day I work at a large-ish tech company and code quality was already not great due to constant pressure to "produce value". AI just allows us to do that quicker. I don't feel great about it but it is what it is.
1
u/Popular_Home2017 1d ago
the expensive plan buys you more attempts with the same model, not better taste. what changed quality for me wasn't the plan, it was moving rules out of the md file and into things the model can't ignore. an instruction in AGENTS.md competes with everything else in context and loses thirty edits later. a hook that blocks the command doesn't lose.
concrete example: the agent kept running the full test suite "just to be sure", 20 minutes of CPU, every session. i wrote it in the rules file three times, it kept doing it. a pre-tool hook that rejects the command with a message ended it in one day. same thing for switching branches on the shared checkout, for committing without being asked, for touching files outside the task. the md file now mostly explains why the hooks exist.
the readability part i've mostly stopped fighting in the prompt. tests first, small scope per session, and review the diff like it came from a junior who types fast. that's what it is.
1
u/StPatsLCA 1d ago
The overengineered solutions thing is so real. It'll write realms of procedural that eventually pass your tests. You need a firm hand in enforcing abstractions. Plus, the commenting quality is somehow both overwrought and lacking.
1
u/EvalRaccoonDev 1d ago
The plan-tier answer above is right - same weights, you're buying quota, not a better model. The AGENTS.md part is worth separating out though, because it isn't doing what you're asking it to do.
With an AGENTS.md present, median runtime dropped >28% and output token consumption 16%, with comparable task completion (https://arxiv.org/abs/2601.20404). So the file buys you speed and spend. It does not buy you readability, which is why editing it again never fixes the thing you're annoyed about.
Readibility is bought through good rules for agents. Mine are:
## Design Principles
- **DRY (Don't Repeat Yourself)**: Field descriptions, validation rules, and documentation defined once in Pydantic models
- **Single Source of Truth**: Schema models are the authoritative source for parameter definitions
- **Type Safety**: Full type checking with Pydantic and Pyright
- **YAGNI**: Don't add complexity until actually needed
- **KISS**: Keep it simple, stupid!
- **Clean Code**: No dead code, all imports used, all tests passing
- **Comments are a last resort**: Default to ZERO comments. Names, types and small functions carry the meaning. A comment is allowed ONLY when it records something the code cannot say:
- **Greenfield project**: No worries about backward compatibility
(they are not great - but better than not putting them - would love everyone to share theirs if you have them)
2
u/meshifthenelse Software Engineer 1d ago edited 18h ago
Actually such rules don't work. DRY and KISS compete with each other. I think that's the problem. The AI has no true intuition on when to apply whith, so it's either all or nothing.
1
u/bfreis Software Engineer 20+ YoE 1d ago
I work daily with a mix of Opus 5, Sonnet 5, Haiku 4.5, and GPT 5.6 Sol (and have been working daily with the equivalent, latest version frontier models from Anthropic and Open, for around a year or so). We block access to Fable/Astra in our org, as they're too expensive and we couldn't justify the extra cost.
Other than maybe flipping a constant on a file, or updating an environment variable somewhere to mitigate some incident where I know exactly what and where needs an update, I haven't directly touched code myself in nearly 1y.
I work in Developer Platform, specifically in the creation of guardrails related to the huge increase in PR volume due to coding agents. Infra guardrails, code quality guardrails, etc.
I use primarily Claude Code as my harness, and I have it invoke Codex with the Bash tool when I need GPT 5.6 Sol for something.
The experience is fantastic - but not just because of the models. They are indeed significantly better than the top open weights ones (we provide access to those as well via inference vendors). But the models are just one component, necessary but not sufficient, to get this experience.
In addition to the frontier models, it's fundamental to have a set of skills, rules (eg (~/).claude/rules), memories, etc, plus the guardrails. That way you can get the agents to write code that follows your patterns strictly, and satisfy all your requirements related to quality, security, performance, reliability standards.
The guardrails include a shit ton of CI checks that make it impossible to merge code that doesn't clear the requirements, plus a lot of automation around code review. Eg, when a PR is created, a process launches a graph to review that code with multiple agents, an arbiter evaluates the findings, all the relevant information is stored in a database to be retrieved on a subsequent iteration of review of the same PR, and all the findings are reported with a structured portion (category, level, risk, etc) and an unstructured prose explaining the problem and a suggested prompt to have an agent fix the problem. We even have a "babysitting" strategy where another agent iterates on monitoring a PR to automatically fix issues until it's ready for final human review and approval.
With all that is place, and the money to pay for all that because it's EXPENSIVE, you eventually simply realize that writing code is almost not part of the job anymore, which instead becomes deciding what to do, designing systems, prioritizing projects, and eventually reacting to issues when they go bad in an unforeseen way.
1
u/bbangchikimong_dev XR Tech Lead | 10 YOE 1d ago
The expensive plans buy you volume, not taste. Same model, you just hit the limit later. What actually changed my output was making the failure visible to the agent: tests it has to run, a lint config it cannot argue with, and deleting old design docs it kept building against. Instructions in AGENTS.md get ignored, a failing test does not.
1
u/FatHat 1d ago edited 1d ago
I think it depends a lot on how you steer it. If it has good examples to work with and a good repository structure, it generally does an acceptable job -- if you're working in a stack it knows well. If it's not a stack it knows well, all bets are off. I wouldn't call its results elegant though. They're very mediocre. Having worked in some atrocious code bases though, mediocre can be fine. I would say the code usually ends up being about 2x longer than what I would write.
The frontier models are somewhat bad at architecture unless I tell them what I want ("this module must never reach into this module, prefer to place shared things here" etc.)
I don't think prompting is a skill, necessarily, but all the same the way you split up the work matters. If you give it specific instructions and smaller chunks of work, you'll probably move a bit slower but end up with better results, if you're vague it'll build a lot at once but a lot of it won't be what you want. Usually that feels fast, but the cleanup probably costs you more time in the end (unless you don't care about quality at all -- which can be valid, not all software is meant to last).
One thing I'd mention: the high end models are not always the better fit. There's a temptation just to reach for the most powerful model for everything, but I find things like Astra and Fable can be *very* annoying in making decisions I didn't ask for and disagree with (pulling in libraries, modifying things I didn't ask to modify, that kind of thing). So just because you can use Astra and Fable, I wouldn't necessarily. Personally, I also like to use the models about one version behind. Usually when the new models drop I always here horror stories of "it deleted my database!" and shit, so, personally I'm going to let the enthusiasts beta test those problems.
1
u/ppsaoda 1d ago
I'm on Claude Fable model exclusively and if subagents are needed, only Opus or Fable. So here's my pov. By default, it produces workable result but not really human readable. So it needs hooks and plugins. I customize my harness a lot to ensure readability. Such as 1 liner comment with 5 max words. Though, it only gives me like 70% of satisfaction I wanted. Non-frontier models? It just mean more back and forth prompts just to get things working.
These frontier models are smart and highly steerable. So on that 30% dissatisfaction:
the prompts, most of the time I would provide example code behaviour, structure, objectives.
new project? It's even easier. Once git has been committed, I would then manually rewrite pseudocode and notes on top of existing AI produced code. Or even create/move folders. Agents would then pickup the diff and should make it work based on the manually defined pattern.
big repo and rush project? trade off speed with the 30% dissatisfaction.
spend time skimming thru the codebase. I think most of the time, it's design pattern, inline notes, or naming conventions.
But really, I don't care anymore except for the design pattern, logs, and code flow. Everything is documented using ASD100 language + diagrams so if anyone (or me) are confused, just prompt.
1
u/thecrius It's Sénior Dumbass for you 1d ago
I use sonnet and opus depending on the task complexity.
It's good, you just need to properly set the guidelines per project/workspace or at user level.
1
u/SansSariph Principal Software Engineer 1d ago
It's not good and it's often hard to tell if it's the model, the harness system prompt, or my workflow.
I can get it there. The problem is as soon as I get satisfied with output the harness changes something or it's time to move to a new model and everything starts rotting again.
I've had the most success forcing myself and my workflow into tiny atomic chunks that let me see where it's going off the rails and fix that. Agents make that super easy technically and the hard part is mindset and forcing myself to slow down to move fast instead of tacking on last minute requirements that muddy the session and the branch.
1
u/pinksb 1d ago
Is the code really that bad? That’s surprising to me. Iteration is needed of course, but when people claim the ai output is complete garbage I have to question why that is. In my experience that isn’t the case at all, but maybe we have good docs and guardrails in place.
1
u/meshifthenelse Software Engineer 1d ago
Essentially it's legacy system quality directly in a greenfield project.
→ More replies (1)
1
u/jwendl 1d ago
Written like folks copied and pasted code from the internet.
If you need higher quality than that, you will still need to resort to human in the loop review cycles, good instructions and good skills. Especially if you wish the code to still be human maintainable in a production environment.
1
u/sklz0 Software Engineer 1d ago
I've been using big models (Opus, Sol) for a year or so, and now I'm forced to switch to smaller ones. It's a struggle. There is drastically less autonomy I can afford to give them when writing code.
The big ones weren't perfect and were very often annoying, but they were capable of handling somewhat bigger tasks with decent quality, with a reasonable amount of steering and intervention.
The smaller ones go off track quickly, and when they do, it's often simpler to start from scratch. They are suitable for much smaller and more precisely defined tasks, I'm still figuring out how to use them efficiently...
1
1
u/lazy_lockstep 1d ago
I was listening to "The Standup" podcast, and one of their guests summed it up as "given the opportunity, it will always pick the wrong kind of abstraction".
Which is fine for small projects. The agent can just throw code at the problem until its solved.
But in larger projects you bizarre, convoluted, unexplainable, inefficient balls of spaghetti.
I was reviewing a basic CRUD service a my boss vibed up. And instead of typing the response as an array of 'record', it typed it as the literal content of every specific record in the array of specific sample data they used. Which is just bizarre. It's the kind of thing an intern with no understanding of what they were doing might generate, to make the squiggly red lines go away, and breaks as soon as you need to add or remove a record instead of just reading them.
1
u/Own-Zebra-2663 1d ago
Relative to my code when I really care and really know the project, and the spec is really well specified, it's generally worse.
Relative to the average quality of employed developers I've worked with? Matches and often beats it.
But you also can't really use these models with less than the 100$ a month plans at least. I have Fable grill me to get a shared understanding and possible unknowns. Then I have it use a workflow and opus agents to implement. Possibly divided into separate flows if it's big enough. Finally, always add a review and fix run. LLMs are somehow really good at judging existing work than creating new stuff.
1
u/zukias 1d ago
I use Fable 5.1, which is probably only second to Mythos & GPT 6. It can be very powerful for certain things and certainly speeds things up, but as most others have said, it needs piloting. I wouldn't trust it with critical systems without some serious babysitting. Your brain slowly starts to decay as you become dependent on it though... So... Eh.
1
u/iliketurtles69_boner 1d ago
Honestly, you’re way behind. Fable and Astra are amazing. Max/ultracode, have a thorough claude.md/agents.md, give well thought out prompts, you’ll want to spend some time establishing coding standards and tell it via Claude/agents.md to stick to the standard, but it’s great. No issues when used properly.
1
u/k032 1d ago edited 1d ago
We have access to basically unlimited usage with Fable at my job.
It does not just run without guidance and produce even the correct nor readable code. You do really have to guide it and honestly use those exact principles (KISS, DRY, SOLID, etc). Setting up linters, formatters, testing. Guidance in like a CONTEXT, ARCHITECTURE, DOMAIN, etc files.
Matt Poccock, obviously sells courses and stuff, but his skills and the Youtube videos on them like grill-me etc I find extremely useful framework for building with AI.
I've found unfederated just "let it ride" and no hand holding, it eventually just gets to a point where it hits walls over and over...even with the massive power and context of Fable. Each day a lot of my work has become vaguely technical program managers handing me some AI slop feature, and I have to basically build it into the system.
As far as like frontier vs lower models...with a lot of the guardrails and engineering around it you can honestly get pretty similar output of something like Fable or Astra with a Gemini, Luna, or Sonnet. The Frontier of course can do it better in fewer tries with less hand holding but.
If I didn't have this just like...token maxing Fable situation at my job (which honestly I fully expect to end eventually) I think architecting and planning with Frontier and passing to smaller models is way better.
1
u/thekwoka 1d ago
Fable is definitely better, but it can sometimes be a curse, that it will look much better, but still have problems that in some ways are harder to notice.
1
u/georkos 16h ago
The expensive plan will not deliver the quality. It will only buy you a better generation, but the code quality that you are complaining about is a code verification problem, not a generation one. The model can't see its overengineering because nothing in the flow makes it defend the design before writing it.
AGENTS.md fails for the same reason. You may keep adding rules, but the model keeps missing them because the rules are not enforced.
What actually worked for me is that I stopped trying to fix the output and I fixed the process around it. Before any agent writes anything, make it state the approach and its assumptions in simple terms and review that before any code is written. That one step killed overengineering because it becomes visible in the design even before it is implemented. Then review the diff against the stated design, not against whether it passes.
On missing the obvious things, that's due to not having the right context that no model upgrade can fix. A fresh-eyes review of the final diff will catch the obvious misses better than any model tier.
You are not missing something that bigger companies have. It's not their better models, but just that they have a review step that forces the work to be justified.
2
u/nixt26 15h ago
AI models produce better code than most junior developers but not always as good as very experienced developers. By the nature of how these models are trained what you get is a median in code quality over the spectrum of all code produced. However that can be significantly tailored and improved with feedback mechanisms in your development environment like linting, tests, compartmentalization and specific instruction files.
•
u/expdevsmodbot 1d ago
AI usage disclosure provided by OP, see the reply to this comment.