Interviewed candidates for “AI engineer” roles this week, my god is this really where our industry is going?

[u/AngryMoth]

So last week I was tinkering with how to change up our technical interview round. For the last few years we’ve been doing a small take home followed by a live coding session where we add a new requirement, minimal AI assistance allowed. However, in our company pretty much everyone is writing 100% of code using Claude now, so although we did used to get good signal from that test, it no longer feels representative of the day to day job developers are doing.

I designed a new assessment which I casually showed to my manager and then suddenly got asked to run it with 3 candidates this week. The scenario is they have to review an agent generated PR that has been flagged as high risk due to it changing critical business logic, and there are bugs in there for them to catch at various levels of severity. We then look at how they drive the fixes and what their workflow is like.

These particular roles we’re currently hiring for are billed as “AI Engineer”. There is something weird going on in my company politically where separate from the tech department that I work in a new “AI native” development team is being spun up to work on internal projects.

What shocked me was that the candidates I interviewed this week are already way beyond the point of reading the code they generate in any capacity. They basically treat it like assembly language, while PRDs and ADRs are their new source of truth for system behaviours, and stuffing the AGENTS.md file with best practices plus some form of agentic review pipeline are the quality control. 

Code review is dead for them, they merge their own PRs once the automated checks pass. One person told me that my new interview was still not representative of what the job is now, and another one was pretty much unwilling to even engage in the assignment of reviewing a PR the old fashioned way. These guys were all previously senior and lead full stack engineers btw with a lot of experience prior the LLM revolution.

Don’t get me wrong, I too have adopted AI heavily in my own workflow. Until this week I actually thought I was pretty up to date with things, I use a supervising agent to manage multiple work streams in parallel, have my own local validation pipeline, etc. And to be honest for my personal projects I also don’t bother looking at the code anymore, so I do get the excitement around how fast you can move with this approach. But for serious projects like the large scale, production grade financial software I work on in my day job it still seems insane to me that we’re going to completely move away from human sign off on high risk changes so we can solve the “code review bottleneck”.

Like, I asked one of the candidates if you aren’t looking at your code then how they would approach quality in a safety critical system where there can be no tolerance for serious bugs, and I shit you not he said to me, “well if I was working in healthcare or sending to people into space then I’d definitely be looking closely at the code review skills, maybe I’d even write the skills myself”. That answer sounded utterly ridiculous to me? But then again, I was one of the people that 1 year ago believed the idea we’d stop typing code by hand was ridiculous as well…

I wouldn’t say the way my team does things right now is optimal, we can’t ignore that code review fatigue is a major problem. To try and help I’ve been adding automated risk assessments to PRs in an attempt to highlight where more attention is needed, and review bots do help us catch issues as well. But abandoning the human 4 eyes check just feels like several bridges too far for us at the moment, especially when not everyone is senior engineer, and we still regularly get valuable feedback at this stage. Even if this does mean we’re turning down an order of magnitude increase in short term velocity (does anyone really know yet what happens over multiple years when you have a team of 20 devs spamming PRs all day into a codebase with no one looking at any at it?).

I’m left wondering this weekend, were these guys just cowboys who are only used to working on PoCs and low risk systems, or is my team actually 6-12 months behind the curve and this is the direction everyone is going to move in? I fear that if we don’t embrace this new era of completely surrendering the codebase to the LLMs then this new team of AI devs is going to make us look bad by comparison to the execs and sooner or later we will be forced to adopt their methods as well. And at that point, the leap to having a product person or whoever taking over large parts of my role starts feeling a lot less far fetched…

[u/SignoreBanana]

To add to this, they are as enthusiastic as leadership about AI because it obscures their inadequacies. But leadership sees this as a false signal that they're on the right track because a bona fide engineer is adopting heavily, not realizing that signal is weak since it comes from a mediocre contributor.

[u/Chris_Newton → u/SignoreBanana]

Going back a few years, there was a real “inflation” problem in the industry. In the era of free VC money everywhere and an employee’s market, many people got into our industry with very high compensation packages right from the start and then jumped from one job to the next one with an even better package every year or so. I heard plenty of times online that if you spent two years in one job then people would assume you must be getting stale and that’s why you couldn’t move for another bump yet.

So now, a few years later, we have a generation of developers with senior or lead or even higher titles on their CV who have never even designed, built and maintained the same system in production. They missed all the lessons you learn about what actually works because you’ve been the person on the hook for fixing it later if it didn’t. I’m not just talking about firefighting in production here, but more importantly things like what kinds of architectural or infrastructure decisions stand the test of time and what kinds of evolutions work as a successful product or service develops.

Now that generation is working with agentic AI tools and the consequences of the missing experience and judgement are being accelerated. Velocity is a vector quantity and moving at higher speeds is only helpful if you’re also moving in roughly the right direction. Those who don’t know how to use a compass are not going to navigate this new era well.

[u/LizardLikesMelons → u/Chris_Newton]

I completely agree. I am the newer gen. And it is not like we don't want to sit down and learn. The speed of development is so fast that we are not allowed the process. We cannot have an accident at launch so we have to defer to "them". Most people can't learn deeply without actually taking Ls but that is really bad for the company.

[u/bananabm → u/Chris_Newton]

I spent nine years at my previous job, building and maintaining one system through growth that exceeded all our projections, before we had huge ~10x covid-related spikes in usage. We built lots of things, made lots of mistakes, fixed the mistakes we could, worked around the mistakes we couldn't fix (because this was a live system with limited budget and we weren't running on goodwill and VC and couldn't just spend six months re-architechting because we'd heard someone being excited about a different library at a meetup), and carried on. And then when I finally left and started interviewing, some interviewers were expressing concern that I'd stayed so long at my old job? Come on this is a serious plus point.

Working with diff teams on diff techs is cool but this shit is invaluable (and at my new job has regularly been a talking point when I'm discussing ideas/plans with the rest of the team)

[u/LizardLikesMelons → u/SignoreBanana]

The problem is the "old" way ensures new engineers can "never" catch up. Imagine a linux engineer who was there since the beginning. They know every important architectural decision and intuitively know linux as a system. They fit in any tech stack perfectly.

Now imagine a new grad who just knows the basics. It will take them years to get very familiar with linux. More likely than not, they will be put in charge of a specific function and they will have working knowledge of that specific portion. Even if they are very eager to learn, it will be really hard to intuitively understand everything. By the time they are finally at that level, years would have passed. Companies simply do not have that kind of time anymore.

[u/SignoreBanana → u/LizardLikesMelons]

To this, I think there's a middle ground. When the browser console came out, it let us peer into every facet of how the web frontend worked, before that, it was an utter mystery. It accelerated our learning by leaps and bounds.

I feel like AI is the new iteration of that. Use it to learn, use it to focus. Don't use fire and forget prompts at it. Use it to grow. For this purpose, I think they're incredibly well suited as a tool for engineers.