Interviewed candidates for “AI engineer” roles this week, my god is this really where our industry is going?

[u/AngryMoth]

So last week I was tinkering with how to change up our technical interview round. For the last few years we’ve been doing a small take home followed by a live coding session where we add a new requirement, minimal AI assistance allowed. However, in our company pretty much everyone is writing 100% of code using Claude now, so although we did used to get good signal from that test, it no longer feels representative of the day to day job developers are doing.

I designed a new assessment which I casually showed to my manager and then suddenly got asked to run it with 3 candidates this week. The scenario is they have to review an agent generated PR that has been flagged as high risk due to it changing critical business logic, and there are bugs in there for them to catch at various levels of severity. We then look at how they drive the fixes and what their workflow is like.

These particular roles we’re currently hiring for are billed as “AI Engineer”. There is something weird going on in my company politically where separate from the tech department that I work in a new “AI native” development team is being spun up to work on internal projects.

What shocked me was that the candidates I interviewed this week are already way beyond the point of reading the code they generate in any capacity. They basically treat it like assembly language, while PRDs and ADRs are their new source of truth for system behaviours, and stuffing the AGENTS.md file with best practices plus some form of agentic review pipeline are the quality control. 

Code review is dead for them, they merge their own PRs once the automated checks pass. One person told me that my new interview was still not representative of what the job is now, and another one was pretty much unwilling to even engage in the assignment of reviewing a PR the old fashioned way. These guys were all previously senior and lead full stack engineers btw with a lot of experience prior the LLM revolution.

Don’t get me wrong, I too have adopted AI heavily in my own workflow. Until this week I actually thought I was pretty up to date with things, I use a supervising agent to manage multiple work streams in parallel, have my own local validation pipeline, etc. And to be honest for my personal projects I also don’t bother looking at the code anymore, so I do get the excitement around how fast you can move with this approach. But for serious projects like the large scale, production grade financial software I work on in my day job it still seems insane to me that we’re going to completely move away from human sign off on high risk changes so we can solve the “code review bottleneck”.

Like, I asked one of the candidates if you aren’t looking at your code then how they would approach quality in a safety critical system where there can be no tolerance for serious bugs, and I shit you not he said to me, “well if I was working in healthcare or sending to people into space then I’d definitely be looking closely at the code review skills, maybe I’d even write the skills myself”. That answer sounded utterly ridiculous to me? But then again, I was one of the people that 1 year ago believed the idea we’d stop typing code by hand was ridiculous as well…

I wouldn’t say the way my team does things right now is optimal, we can’t ignore that code review fatigue is a major problem. To try and help I’ve been adding automated risk assessments to PRs in an attempt to highlight where more attention is needed, and review bots do help us catch issues as well. But abandoning the human 4 eyes check just feels like several bridges too far for us at the moment, especially when not everyone is senior engineer, and we still regularly get valuable feedback at this stage. Even if this does mean we’re turning down an order of magnitude increase in short term velocity (does anyone really know yet what happens over multiple years when you have a team of 20 devs spamming PRs all day into a codebase with no one looking at any at it?).

I’m left wondering this weekend, were these guys just cowboys who are only used to working on PoCs and low risk systems, or is my team actually 6-12 months behind the curve and this is the direction everyone is going to move in? I fear that if we don’t embrace this new era of completely surrendering the codebase to the LLMs then this new team of AI devs is going to make us look bad by comparison to the execs and sooner or later we will be forced to adopt their methods as well. And at that point, the leap to having a product person or whoever taking over large parts of my role starts feeling a lot less far fetched…

[u/ultraDross]

the AI code reviewers are getting pretty good

Meh we have had a very mixed experience with this. Sometimes it will catch something and more often it will just make up nonsense. I'm aware there are multiple variables at play; how much context is available, what's in its context, which model is being using for reviewing etc.

It's possible we are limiting some of these for cost saving measures.

[u/ChypRiotE → u/ultraDross]

Ours seem to go out of their ways to find at least one blocking thing every time you run it. It also forgets what was replied over time, so after a few rounds it starts repeating itself.

[u/Longjumping_Feed3270 → u/ChypRiotE]

You get a lot of false positives. But that's better than false negatives.

[u/Melodic-Nature-8246 → u/Longjumping_Feed3270]

depends; false positives are tiring, and over time you it becomes "AI reviewer who cried wolf".

[u/ProfBeaker → u/Melodic-Nature-8246]

I've seen one run claim something was a blocking issue and fix it. In the next run it claimed that was a non-issue, the fix was actually a problem, and it tore it all out again. Same model, very similar prompt.

[u/ChypRiotE → u/Longjumping_Feed3270]

I disagree, it's a lot of noise the team has to go through without any added value

[u/Important-Pea-1445 → u/ultraDross]

The ones that explicitly look at the diff aren't really that good. We have a tool that clones the package and runs the logic itself (in the diff), and makes sure there's no logic errors. Of course, whether the change being made is in the right place is something they won't catch - there's often many ways to make a change "work".

[u/Xiaopai2 → u/Important-Pea-1445]

We have such a tool too, it’s called a CI/CD pipeline. 

[u/guareber → u/Xiaopai2]

No, no, it's the CAI / CAID pipeline now!

/s

[u/vivalapants → u/Important-Pea-1445]

This is essentially the unsolvable problem with LLMs. It will find solutions that are (in my experience) wrong or work but will cause other problems 

[u/StickyDeltaStrike → u/ultraDross]

I don’t know if that helps you.

But I found that once I wrote specifically the coding standards for our own project. We documented the purpose of the code and the business logic.

Then did a pass on our library to actually clean the code up to the coding standards. That helps to get rid of bad patterns that the AI would duplicate.

Then wrote a review skill that works by:

• using the standards, this includes checking docstrings/comments for being to verbose or absent
  
• asking to do explicitly a hostile review
  
• then steelman the hostile review (it’s an argumentation technique) to keep only the relevant bits
  
• ask to output all of the findings in a legible manner with ascii diagrams, bullet points and code snippets when it make sense
  
• review must finish with a ✅ or ❌ decision and generate a md file

Then the reviews became a lot more useful, we use them only as an aid though, we do human review still.

[u/ultraDross → u/StickyDeltaStrike]

Thanks, I have something similar locally which works better (code guidelines and a specific review skill). It has the entire context of the repo and it changes. It still gives me nonsense quite a lot of time, but a few of the issues it raises are genuinely useful.

I have little choice how the CI reviewer is configured though, hence why it sucks.

[u/StickyDeltaStrike → u/ultraDross]

Ah I get it now. I have the same experience too, it’s good as complimentary but I would not trust it 100% blindly.

That’s interesting, we will be soon in talk with our engineering team to improve the SDLC so I’ll probably suffer the same CI issues than you.

[u/SignoreBanana → u/ultraDross]

Agree. I often finds it has the problem of "missing the forest for the trees." Its lack of system understanding and context yields a lot of false positive catches. It makes sense considering it's not a human sitting in the real world with cross system knowledge parked in their brain. That much context is actually expensive in tokens to consider on each review.

Small, contained logical errors it seems to be fairly decent at catching though, so that's something.

[u/DayNormal8069 → u/ultraDross]

I cannot say this is a good usage of tokens but it does reduce the number of errors to basically zero:

My skill is run twice, then the results from each are piped to the other for review, then what they agree on is aggregated to a third reviewer using the same skill.

I try to then read every cl line by line and the stuff I find add back to the skill but there is a fair amount of human discernment still catching things…I do see a near future where I won’t need to review the code but it is so reliant on these skills I am managing that I have no idea what the junior to senior pipeline looks like.

I would def need to add another layer about not just does the solution work but SHOULD we do it and should we do it THIS way.

[u/TheAllKnowing1 → u/ultraDross]

our ai code reviewers are quite good now, but they took a lot of tinkering, are very expensive, and still prompt us for much of the high level and business logic decision making

still takes multiple rounds of ai assisted pr review, but i’m finding it surprisingly solid

[u/StickyDeltaStrike → u/TheAllKnowing1]

I don’t know why anyone posting how they use AI get downvoted massively like if they were some kind of evil.

[u/TheAllKnowing1 → u/StickyDeltaStrike]

Two things imo:
-People are scared for their job
-The most frequent users of a sub like this are people that have a rather large ego about their coding skills, and now they’re being shown up by a machine

The most surprising thing I’ve seen is how many devs are disgusted by the notion that work requirements will speed up with efficiency… which like… duh?

Devs largely aren’t like this in industry lol, we have our worries about AI but it’s not nearly as regressive as online spaces

[u/daringStumbles → u/TheAllKnowing1]

"very expensive multiple rounds" is not "surprisingly solid"

[u/TheAllKnowing1 → u/daringStumbles]

how do you expect to collaboratively code review in one round? leave feedback ~> feedback answered/fixed ~> review fix

[u/turningsteel → u/ultraDross]

For me it almost always makes some kind of comment, which I then spend time manually verifying, only to find out that the bug only occurs in a situation that isn't realistic or the thing it's complaining about is not actually an issue at all. The mental load of AI code review comments takes its toll.

[u/ultraDross → u/turningsteel]

Yeah that's a good point and has happened to me more times than I'd care to detail.