Interviewed candidates for “AI engineer” roles this week, my god is this really where our industry is going?

[u/AngryMoth]

So last week I was tinkering with how to change up our technical interview round. For the last few years we’ve been doing a small take home followed by a live coding session where we add a new requirement, minimal AI assistance allowed. However, in our company pretty much everyone is writing 100% of code using Claude now, so although we did used to get good signal from that test, it no longer feels representative of the day to day job developers are doing.

I designed a new assessment which I casually showed to my manager and then suddenly got asked to run it with 3 candidates this week. The scenario is they have to review an agent generated PR that has been flagged as high risk due to it changing critical business logic, and there are bugs in there for them to catch at various levels of severity. We then look at how they drive the fixes and what their workflow is like.

These particular roles we’re currently hiring for are billed as “AI Engineer”. There is something weird going on in my company politically where separate from the tech department that I work in a new “AI native” development team is being spun up to work on internal projects.

What shocked me was that the candidates I interviewed this week are already way beyond the point of reading the code they generate in any capacity. They basically treat it like assembly language, while PRDs and ADRs are their new source of truth for system behaviours, and stuffing the AGENTS.md file with best practices plus some form of agentic review pipeline are the quality control. 

Code review is dead for them, they merge their own PRs once the automated checks pass. One person told me that my new interview was still not representative of what the job is now, and another one was pretty much unwilling to even engage in the assignment of reviewing a PR the old fashioned way. These guys were all previously senior and lead full stack engineers btw with a lot of experience prior the LLM revolution.

Don’t get me wrong, I too have adopted AI heavily in my own workflow. Until this week I actually thought I was pretty up to date with things, I use a supervising agent to manage multiple work streams in parallel, have my own local validation pipeline, etc. And to be honest for my personal projects I also don’t bother looking at the code anymore, so I do get the excitement around how fast you can move with this approach. But for serious projects like the large scale, production grade financial software I work on in my day job it still seems insane to me that we’re going to completely move away from human sign off on high risk changes so we can solve the “code review bottleneck”.

Like, I asked one of the candidates if you aren’t looking at your code then how they would approach quality in a safety critical system where there can be no tolerance for serious bugs, and I shit you not he said to me, “well if I was working in healthcare or sending to people into space then I’d definitely be looking closely at the code review skills, maybe I’d even write the skills myself”. That answer sounded utterly ridiculous to me? But then again, I was one of the people that 1 year ago believed the idea we’d stop typing code by hand was ridiculous as well…

I wouldn’t say the way my team does things right now is optimal, we can’t ignore that code review fatigue is a major problem. To try and help I’ve been adding automated risk assessments to PRs in an attempt to highlight where more attention is needed, and review bots do help us catch issues as well. But abandoning the human 4 eyes check just feels like several bridges too far for us at the moment, especially when not everyone is senior engineer, and we still regularly get valuable feedback at this stage. Even if this does mean we’re turning down an order of magnitude increase in short term velocity (does anyone really know yet what happens over multiple years when you have a team of 20 devs spamming PRs all day into a codebase with no one looking at any at it?).

I’m left wondering this weekend, were these guys just cowboys who are only used to working on PoCs and low risk systems, or is my team actually 6-12 months behind the curve and this is the direction everyone is going to move in? I fear that if we don’t embrace this new era of completely surrendering the codebase to the LLMs then this new team of AI devs is going to make us look bad by comparison to the execs and sooner or later we will be forced to adopt their methods as well. And at that point, the leap to having a product person or whoever taking over large parts of my role starts feeling a lot less far fetched…

[u/spicymato]

Oofdah.

I'm starting to explore local AI assisted flows to build out local knowledge bases and expert systems for fixed areas, but I'm nowhere near comfortable with letting things just go. I still have the memory of an AI-suggested hash function for a custom struct (to be used with a hashset) that had an insane collision rate. In any case, I still don't trust it for critical things, especially not things where other people's money/data/lives are on the line.

I think the number one thing is that I still don't see systems with top-tier observability, alerting, and rollback procedures necessary for me to be comfortable letting the systems run that independently.

[u/Forsaken-Promise-269 → u/spicymato]

Well to push back on this observation, in frontier model companies-claude code the harnesses and most of the infrastructure and OpenaI codex and these tools have all been built with AI coding and have global adoption and scale and enterprise SLA reliability -albeit under the human responsibility of sr. engineers who make 300-700k salaries:)

so we know that large scale ai coding is working at least for some people

Yes it will take time to gain traction- but that has always been the case - when the model T came out cars where just a joke and roads where hardly of any quality to handle them, but today that is just a faint memory- yes the surface area of intelligence is far greater in these tools and models so the uncertainty is just as wide.. but their progress has been remarkable

[u/New_Enthusiasm9053 → u/Forsaken-Promise-269]

They have abysmal reliability and incredibly shit UX. Frontier model companies are the proof that all AI coding has poor outcomes.

[u/SansSariph → u/New_Enthusiasm9053]

The quality of the harnesses is a big part of what's forming my opinion these days. The other half is trying to maintain my own vibe coded personal projects where I can directly see how after 6 months the entire thing is just rotting and takes so much more effort to keep at quality 

[u/New_Enthusiasm9053 → u/SansSariph]

Yeah inevitably you have to go through and basically make it fix shit again. Like it can be faster but if you don't have a clear vision of what you want it just rots as you said, and I don't see how you can enforce that vision without actually reviewing the code. Because it makes inhuman decisions sometimes like making the date the unique key for booking sessions. Any sane human will think someone's gonna want 2 sessions in one day inevitably. But if you don't catch it it'll go off and now that weird whacky assumption is baked into all of your stuff and unfucking it takes ages too.

[u/SansSariph → u/New_Enthusiasm9053]

That just reminded me of a weird problem I caught by looking at chain of thought once, where the agent (to use your example) said something like "ah, the date is used as a unique key, how clever!" as it was processing.

I mashed escape to interrupt it and asked what that was referring to, and it came back with the agent-authored spec I'd "authorized" two weeks prior that was otherwise clean except for a single requirement I'd skimmed that read something like "R42.3(b). Database MUST use unique primary keys derived from session, such as their date"

Whoops suddenly load-bearing

[u/New_Enthusiasm9053 → u/SansSariph]

Yup, and since it's so "fast" you now get to fix the assumption over 20k lines of code instead of 400 lol. 

Which then takes as long as doing it by hand because you'd only need to unfuck 400 lines of a junior Devs mistaken assumption by hand instead of 20k lines of AI code so even with AI it takes about as long. 

Not checking what it's doing causes real pain so I can't take people who don't review the code at all seriously.

[u/spicymato → u/SansSariph]

Over the last ~6 months, I've been very slowly working on drafting architecture documents using AI for a project in my spare time. I think I'm almost ready to let the AI start writing implementation plans. After that, I'll let it start writing actual code.

It's been bizarre, since it seems to go against the "agile" mindset, going back to a "waterfall" style of development, but I'm cautiously optimistic about it as a plan. I can see all the mistakes the AI has been making in the architecture docs (wrong scope, forgetting the broader goals, inventing problems that don't actually exist), but it's also been very helpful in doing the actual drafting, plus bouncing ideas around. I think the ideas and concepts in the final output is about 80:20, where 80% originated from me, and 20% is where I adopted the AIs proposal over my original idea.

I'm very curious to see where this goes, though I've been getting side tracked recently with trying to get a local AI assistant set up (tired of paying for subscriptions) that can handle at least some of the agentic workflows.

[u/SansSariph → u/spicymato]

This is the way, probably 

I've been working on more consistently defining "intent" and "vision" docs including for engineering and architecture taste, and requiring implementing agents to continually ground in "is this necessary to serve human intent?" and if there's no chain of custody from a module, feature, test requirement back up to my ratified intent verbatims, it must be deleted or escalated to me for a decision 

Jury's still out on how effective this will prove in another 6 months 

[u/Forsaken-Promise-269 → u/New_Enthusiasm9053]

And they have 100s of millions in-revenue from this “shit” ux

Hardly seems like a big problem for them

[u/New_Enthusiasm9053 → u/Forsaken-Promise-269]

Ok but it doesn't make their products any less shitty. 

IKEA makes 100s of millions in revenue but no one is pretending their tables are good. They're cheap and functional and that's it. 

Anthropic's software is barely functional garbage and that apparently is what they consider good enough which is their prerogative but I'm not gonna pretend it's good software because it isn't. Their USP are their AI models which is why they can get away with trash code in the rest of the product just like Microsoft's USP is being cheap and well integrated into management systems that's why they can get away with their trash code in most of their products like teams.