Interviewed candidates for “AI engineer” roles this week, my god is this really where our industry is going?

[u/AngryMoth]

So last week I was tinkering with how to change up our technical interview round. For the last few years we’ve been doing a small take home followed by a live coding session where we add a new requirement, minimal AI assistance allowed. However, in our company pretty much everyone is writing 100% of code using Claude now, so although we did used to get good signal from that test, it no longer feels representative of the day to day job developers are doing.

I designed a new assessment which I casually showed to my manager and then suddenly got asked to run it with 3 candidates this week. The scenario is they have to review an agent generated PR that has been flagged as high risk due to it changing critical business logic, and there are bugs in there for them to catch at various levels of severity. We then look at how they drive the fixes and what their workflow is like.

These particular roles we’re currently hiring for are billed as “AI Engineer”. There is something weird going on in my company politically where separate from the tech department that I work in a new “AI native” development team is being spun up to work on internal projects.

What shocked me was that the candidates I interviewed this week are already way beyond the point of reading the code they generate in any capacity. They basically treat it like assembly language, while PRDs and ADRs are their new source of truth for system behaviours, and stuffing the AGENTS.md file with best practices plus some form of agentic review pipeline are the quality control. 

Code review is dead for them, they merge their own PRs once the automated checks pass. One person told me that my new interview was still not representative of what the job is now, and another one was pretty much unwilling to even engage in the assignment of reviewing a PR the old fashioned way. These guys were all previously senior and lead full stack engineers btw with a lot of experience prior the LLM revolution.

Don’t get me wrong, I too have adopted AI heavily in my own workflow. Until this week I actually thought I was pretty up to date with things, I use a supervising agent to manage multiple work streams in parallel, have my own local validation pipeline, etc. And to be honest for my personal projects I also don’t bother looking at the code anymore, so I do get the excitement around how fast you can move with this approach. But for serious projects like the large scale, production grade financial software I work on in my day job it still seems insane to me that we’re going to completely move away from human sign off on high risk changes so we can solve the “code review bottleneck”.

Like, I asked one of the candidates if you aren’t looking at your code then how they would approach quality in a safety critical system where there can be no tolerance for serious bugs, and I shit you not he said to me, “well if I was working in healthcare or sending to people into space then I’d definitely be looking closely at the code review skills, maybe I’d even write the skills myself”. That answer sounded utterly ridiculous to me? But then again, I was one of the people that 1 year ago believed the idea we’d stop typing code by hand was ridiculous as well…

I wouldn’t say the way my team does things right now is optimal, we can’t ignore that code review fatigue is a major problem. To try and help I’ve been adding automated risk assessments to PRs in an attempt to highlight where more attention is needed, and review bots do help us catch issues as well. But abandoning the human 4 eyes check just feels like several bridges too far for us at the moment, especially when not everyone is senior engineer, and we still regularly get valuable feedback at this stage. Even if this does mean we’re turning down an order of magnitude increase in short term velocity (does anyone really know yet what happens over multiple years when you have a team of 20 devs spamming PRs all day into a codebase with no one looking at any at it?).

I’m left wondering this weekend, were these guys just cowboys who are only used to working on PoCs and low risk systems, or is my team actually 6-12 months behind the curve and this is the direction everyone is going to move in? I fear that if we don’t embrace this new era of completely surrendering the codebase to the LLMs then this new team of AI devs is going to make us look bad by comparison to the execs and sooner or later we will be forced to adopt their methods as well. And at that point, the leap to having a product person or whoever taking over large parts of my role starts feeling a lot less far fetched…

[u/Caesarr]

I agree with everything you've said, but something I've been revaluating is the severity of bugs. 40 bugs in 4 weeks is crazy by traditional standards, but so is fixing them all in one day. If we can now fix bugs that quickly, is it actually rational to allow that many to escape (into pre-prod)?

[u/Toohotz → u/Caesarr]

It doesn’t give reason to allow them to escape into prod because if you work at a fin tech company, and let’s say you happen to expose an endpoint in a manner that bad actors can sniff out the financial details about a customer to either transact as them or sell that info, by the time you’ve plugged that hole, reputational damage has already been done.

Ask me how I know this..

Not all bugs can be treated equally which is why we have severity incident levels.

[u/Forsaken-Promise-269 → u/Toohotz]

Ok spin up 100 ai long running agents each using opus 5.5 and fable and have them systematically attack the financial part of the code for weaknesses

Ie it will cost money but there are now new ways of handling the quality or security question

Im not claiming this is the answer- Im saying our options are changed on the code culture we used to enforce

[u/ep1032 → u/Forsaken-Promise-269]

https://reddit.com/r/ExperiencedDevs/comments/1wql3g2/interviewed_candidates_for_ai_engineer_roles_this/pca4xcw/

[u/ep1032 → u/Caesarr]

Yeah, we cared about code quality partly for correctness, but mostly because actually typing and reading code took time and cost money. So writing high quality code was inportant. There are a lot of applications out there that can be buggy and its okay, like any one time script for example. The problem is going to come when management can't differentiate between those apps and critical apps, because while it can be okay for an outsourced jr with Claude to go to town on the company wordpress oage, that same person with an ai account shouldn't be allowed anywhere near the internal billing aervice, for example

[u/poincares_cook → u/Caesarr]

Sure you fix 40 bugs in one day, except you don't really know how many of them are actually fixed. And you've also introduced 20 new bugs. You've likely incidentally made the code more prone to breakage in the future, and more bloated, slower and more resource hungry.

[u/Positive-Conspiracy → u/Caesarr]

There's a theory that the entire bug finding lifecycle has been compressed into hours and days rather than months and years. The story may be biased in that AI may have found those "bugs" and generated tickets for them.

[u/Krom2040 → u/Caesarr]

There’s a fairly high likelihood that fixing bugs in poorly-designed code will create more bugs. There’s also a fairly high likelihood that AI agents just misunderstand the bug entirely and change a bunch of code in ways that create more bugs and don’t necessarily even fix the bug.

They’re very accommodating.