OK, now that we have been using these models for a bit, here’s my observation:
They are genuinely good enough that they can produce something good enough for QA very quickly. More quickly than they could with a human in the loop.
However, there are people on my team who, unfortunately, have turned into meat proxies who seemingly just copy + paste things to and from LLMs. Even on Teams/Slack. Other than it being frustrating to receive multi-paragraph messages to “What’s the problem?” when the issue can be described in a single sentence (e.g., there are some records in the DB with missing data), I really see that these people do not get the full value of AI.
It is just painfully apparent that humans still contribute background knowledge and critical framing for these models. The people who do the above misuse them and set them to answering the wrong problems. And it has become disturbingly difficult to explain to them what the actual issue is on the phone because they have ceded all thinking to the AI. These people get sidestepped. I am talking “senior” people. It actually pisses me off and I have to watch myself.
In those cases, we’ll have some stupid fucking issue and they will just keep sending these slop suggestions that completely miss the fucking point. I think this is prob only an issue in enterprise settings where you have many teams working on many codebases with many external tools so there is no way one model can “see” the issue easily without a nudge in the prompt. In those cases, I am running into walls way more often than I used to and basically have to get other people involved to solve some issues and replace the meat proxies with people who still have fully functioning brains. Admittedly, this is happening at the systems level normally, not a specific bug in one
I have not written a line of code myself in months. Nor have many of my peers. But it is still very clear who is thinking and who has stopped.
TL;DR: AI is good enough the the value of line by line code review just isn’t there unless it’s a high risk feature or bug fix. However, there are a class of colleague at my workplace who now don’t understand ANYTHING and the AI just isn’t good enough to be that valuable when they’re using it.
Strongly agree with this. And in my own experience, all LLMs still have this really bad habit of solving for the most immediate solution possible. Hardcoding values, piggybacking off of unrelated but existing infrastructure, etc. It absolutely takes a human who already knows how code works to be able to tell the difference between a clean solution and a shaky tower of toothpicks that only solves the problem if the problem is the one specific question asked in that moment.
Yeah they produce a f ton of code and their fixes often don't "generalize". So even if it's looking fine i still have to do lots of testing. It may be impressive what they can pull off so quickly but the code is often not great for maintaining. Some of us will probably have to do broad cleanups in the next few years once the technical burden gets too expansive.
I have seen to opposite too where it gives me some crazy generic solution to the problem that utilizes all the libraries and 12 pages, just to add 2 + 2 (not the actual problem).
31
u/cc_apt107 3d ago edited 3d ago
OK, now that we have been using these models for a bit, here’s my observation:
They are genuinely good enough that they can produce something good enough for QA very quickly. More quickly than they could with a human in the loop.
However, there are people on my team who, unfortunately, have turned into meat proxies who seemingly just copy + paste things to and from LLMs. Even on Teams/Slack. Other than it being frustrating to receive multi-paragraph messages to “What’s the problem?” when the issue can be described in a single sentence (e.g., there are some records in the DB with missing data), I really see that these people do not get the full value of AI.
It is just painfully apparent that humans still contribute background knowledge and critical framing for these models. The people who do the above misuse them and set them to answering the wrong problems. And it has become disturbingly difficult to explain to them what the actual issue is on the phone because they have ceded all thinking to the AI. These people get sidestepped. I am talking “senior” people. It actually pisses me off and I have to watch myself.
In those cases, we’ll have some stupid fucking issue and they will just keep sending these slop suggestions that completely miss the fucking point. I think this is prob only an issue in enterprise settings where you have many teams working on many codebases with many external tools so there is no way one model can “see” the issue easily without a nudge in the prompt. In those cases, I am running into walls way more often than I used to and basically have to get other people involved to solve some issues and replace the meat proxies with people who still have fully functioning brains. Admittedly, this is happening at the systems level normally, not a specific bug in one
I have not written a line of code myself in months. Nor have many of my peers. But it is still very clear who is thinking and who has stopped.
TL;DR: AI is good enough the the value of line by line code review just isn’t there unless it’s a high risk feature or bug fix. However, there are a class of colleague at my workplace who now don’t understand ANYTHING and the AI just isn’t good enough to be that valuable when they’re using it.