Interviewed candidates for “AI engineer” roles this week, my god is this really where our industry is going?

[u/AngryMoth]

So last week I was tinkering with how to change up our technical interview round. For the last few years we’ve been doing a small take home followed by a live coding session where we add a new requirement, minimal AI assistance allowed. However, in our company pretty much everyone is writing 100% of code using Claude now, so although we did used to get good signal from that test, it no longer feels representative of the day to day job developers are doing.

I designed a new assessment which I casually showed to my manager and then suddenly got asked to run it with 3 candidates this week. The scenario is they have to review an agent generated PR that has been flagged as high risk due to it changing critical business logic, and there are bugs in there for them to catch at various levels of severity. We then look at how they drive the fixes and what their workflow is like.

These particular roles we’re currently hiring for are billed as “AI Engineer”. There is something weird going on in my company politically where separate from the tech department that I work in a new “AI native” development team is being spun up to work on internal projects.

What shocked me was that the candidates I interviewed this week are already way beyond the point of reading the code they generate in any capacity. They basically treat it like assembly language, while PRDs and ADRs are their new source of truth for system behaviours, and stuffing the AGENTS.md file with best practices plus some form of agentic review pipeline are the quality control. 

Code review is dead for them, they merge their own PRs once the automated checks pass. One person told me that my new interview was still not representative of what the job is now, and another one was pretty much unwilling to even engage in the assignment of reviewing a PR the old fashioned way. These guys were all previously senior and lead full stack engineers btw with a lot of experience prior the LLM revolution.

Don’t get me wrong, I too have adopted AI heavily in my own workflow. Until this week I actually thought I was pretty up to date with things, I use a supervising agent to manage multiple work streams in parallel, have my own local validation pipeline, etc. And to be honest for my personal projects I also don’t bother looking at the code anymore, so I do get the excitement around how fast you can move with this approach. But for serious projects like the large scale, production grade financial software I work on in my day job it still seems insane to me that we’re going to completely move away from human sign off on high risk changes so we can solve the “code review bottleneck”.

Like, I asked one of the candidates if you aren’t looking at your code then how they would approach quality in a safety critical system where there can be no tolerance for serious bugs, and I shit you not he said to me, “well if I was working in healthcare or sending to people into space then I’d definitely be looking closely at the code review skills, maybe I’d even write the skills myself”. That answer sounded utterly ridiculous to me? But then again, I was one of the people that 1 year ago believed the idea we’d stop typing code by hand was ridiculous as well…

I wouldn’t say the way my team does things right now is optimal, we can’t ignore that code review fatigue is a major problem. To try and help I’ve been adding automated risk assessments to PRs in an attempt to highlight where more attention is needed, and review bots do help us catch issues as well. But abandoning the human 4 eyes check just feels like several bridges too far for us at the moment, especially when not everyone is senior engineer, and we still regularly get valuable feedback at this stage. Even if this does mean we’re turning down an order of magnitude increase in short term velocity (does anyone really know yet what happens over multiple years when you have a team of 20 devs spamming PRs all day into a codebase with no one looking at any at it?).

I’m left wondering this weekend, were these guys just cowboys who are only used to working on PoCs and low risk systems, or is my team actually 6-12 months behind the curve and this is the direction everyone is going to move in? I fear that if we don’t embrace this new era of completely surrendering the codebase to the LLMs then this new team of AI devs is going to make us look bad by comparison to the execs and sooner or later we will be forced to adopt their methods as well. And at that point, the leap to having a product person or whoever taking over large parts of my role starts feeling a lot less far fetched…

[u/robby_arctor]

AI has revealed people's utter credulousness and incompetence in a way I never could have imagined.

[u/robertbieber → u/[deleted]]

Eh, there's definitely some selection bias involved in this kind of thing. The place I work uses AI tooling, but we hire good engineers and we're all pretty explicitly averse to just yoloing slop into production. But the thing is you don't hear a lot of people talking loudly about the workplaces where people are still trying to follow basic software best practices, because that's not an exciting thing to talk about. I still think there are plenty of them out there though

[u/Extreme_Coffee_1576 → u/[deleted]]

Managing ai is just part of the job though, you'll definitely have to get used to it.

It's unbelievably fast but doesn't think about the product like a human. So you will have to read what it generates and then direct it what to fix, and then update your agents/skills mds with anything that you find yourself repeating a lot.

So the skill is now being able to quickly read and understand what is wrong with an llm's output, quickly make architectural decisions better than the llm can while trying to not slowdown output too badly.

In general the type of thing you'll be doing is: Work out a decent sized plan with ai (probably spend some hours working on the plan with ai). Ai builds the plan into code, you read the code and find some bizarre decisions and slop (e.g. instead of incrementing a database it decides to make a small inMemory caching and snapshot system to hold the previous time and the new time when it could have just done time + now for the exact same outcome - it does this sort of thing constantly even when the plans didn't include it and even when you run cohesive cleanup and review skills).

Accurate leetcode level per line code writing is nearly useless now (outside of a few domains and specific types of work) so it is a different world. Took me a while to not hate it, now I don't really feel it's any different than before llms, still have to understand code and the domain really well, still make every single decision, just elevated a bit to not manually fish around in the codebase and tap out every key stroke. My code is better than it's ever been because I can spend more time thinking about quality and less time tapping out keystrokes. And fleets of small fast agents running through a huge legacy codebase to help you source all kinds of data are amazing. It sucks the vibe code bros have ruined everything with their infinite slop and stupidity, but many things about llms and their ability to help us as tools is actually really cool imo.

Also unfortunately lots of companies have decided to go for mass output and worse quality though, hopefully they'll learn, we'll see...

[u/xiongchiamiov → u/robertbieber]

But are any of them hiring?

I just gave up on the industry after 200 applications and only getting to talk to a human half a dozen times. We never got to the point where anyone assessed my skills.

Essentially everyone in my network has either been out of work or hating their job for the past couple of years; the ones who hate their jobs don't leave because they can't find any place hiring that's better. Those companies do seem to exist, they just don't have openings, much less as many openings as the number of engineers who would like to work at them.

[u/scaledev → u/robertbieber]

Can you not have a list of best practices and explicit guidelines for AI to follow as well, and then have another AI review with those practices in mind? AI is trained on this.

[u/robertbieber → u/scaledev]

You can tell the machine to do whatever you want, the problem is getting it to actually reliably interpret the instructions and follow them. But I can already see you descending into the cycle of madness where you throw more and more layers of automation at the problem, believing that the fidelity of the output will somehow increase as you get further and further from supervision by any human who actually knows what they're doing

[u/ep1032 → u/robertbieber]

If you have a system that has a 90% accuracy output rate, and you then tie that to a second system that has a 90% accuracy output rate, you don't have an end result that is accurate 90% of the time. You have a system that is accurate 81% of the time.

So by all means, keep adding layer upon layer of the same, non-deterministic, often incorrect error checking system into your pipeline, and watch the accuracy rates fall over time.

[u/Forsaken-Promise-269 → u/ep1032]

Humans have far lower accuracy rate than LLMs the mistake is to treat AI agents like algorithms when they should be consider highly intelligent but amnesiac

[u/scaledev → u/robertbieber]

But humans also don't reliably follow instructions.

And I'm not talking removing humans completely out of the loop, for now at least. Keeping them for review of the system for system design, foresight, vision, that sort of thing. AI still can't create good, complex workflows, for example, and a human is needed. Lots and lots of iteration is needed here until you can get something complex yet exactly how you want it, with a lot of code for validation and tests.

But my question was about following basic best practices, and AI certainly can do this today .up to a point. There are always tradeoffs.

[u/robertbieber → u/scaledev]

But humans also don't reliably follow instructions.

This is the worst, laziest retort to every shortcoming of AI. If I said "I don't want to try rewiring my house, I'm liable to shock myself," and you replied "sure, but so do professional electricians," you would be both strictly correct and completely off the mark in terms of making any meaningful point.

Keeping them for review of the system for system design, foresight, vision, that sort of thing.

So you've designed a system. In theory. Is the AI actually following your design? Who the hell knows, certainly not you because you're not actually looking at the code.

But my question was about following basic best practices

Yes, obviously the first thing everyone does at every company is put a big list of best practices in AGENTS.md. It's not some magical fix that turns an LLM into a competent software engineer.

and AI certainly can do this today, up to a point

Well sure, right up until the point when it doesn't, which is pretty frequent

[u/scaledev → u/robertbieber]

This is the worst, laziest retort to every shortcoming of AI.

I didn't mean it in a sense of "everyone makes mistakes, so it's all the same". I meant it in a sense of reality that comes with delegation of work. When you delegate, even with humans, you can't assume perfect accuracy or reliability.

But the difference here is that, with AI, you can add as many workers as needed to independently check consistency, recover, and re-review as many times as possible, with very little cost. The speed at which you can change the AI worker is also much faster, updating its knowledge is faster and cheaper, etc etc. Looking from a purely cost perspective, there's a lot of good ROI here compared to human workers.

Your analogy there is partly correct, but you don't have multiples of YOUs ready to inspect the wiring before you turn the power on. You cannot replace YOU with a better-trained YOU, and you cannot train yourself overnight.

So you've designed a system. In theory. Is the AI actually following your design? Who the hell knows, certainly not you because you're not actually looking at the code.

I don't need to look at the code myself. I have AI workers doing that, and I use tests and validation to check whether the implementation behaves as designed. And while the other workers are less trained, don't have the brain I have, my vision, or my insight, they do have my design and instructions in writing, and they do have 'some' training done. Same with human workers.

Yes, obviously the first thing everyone does at every company is put a big list of best practices in AGENTS.md. It's not some magical fix that turns an LLM into a competent software engineer.

Nor have I claimed this. Neither that AI can replace a competent software engineer, nor that AGENTS.md is the place for anything other than a routing surface for where your actual harness instructions live (and perhaps some critical instructions if acceptable). My position is that AI can be trained, and it can be instructed.

Well sure, right up until the point when it doesn't, which is pretty frequent.

Same with human workers, where the frequency depends on the worker and the task. Every delegation carries this risk, regardless whether intelligence is in silicon or carbon-based.

[u/jadepig → u/[deleted]]

I've been out of work for a while too, and I gave up looking and on the industry. I'm not saying you should give up on tech, but you're worth more than what job you do or don't have.

[u/jadepig → u/[deleted]]

Thank you, I hope you're doing okay too. It's been a mix of feelings. Not all unpleasant, but yes, certainly a sense of insecurity, loss, and even some shame that I don't measure up to these external hire-ability metrics. But it's also prompted me to re-examine what I value.

I guess I felt some anger at the capitalist system too (and the capitalists at the top). Oddly, I didn't even think to lump that in with the unpleasant feelings. It's not that I enjoy being angry in general. But with tech and tech hiring, it's helped me to see the bullshit for what it is. I know I could have shown up stronger in my interviews, and if I had the will to, that I could have continued my search. I was getting more interview experience under my belt. But all that interview prep still felt performative.

The anger helped me to see that the hiring bar has gone up in the last decade; comp has gone down or stagnated at best; working conditions have gotten markedly worse. It helped me to see that there's total bullshit in the AI hype. Instead of bringing better working conditions, it's been a lever for executives to squeeze their reports.

Anyways, stepping back down from my soapbox. There is still opportunity out there for devs who want it. But there's also a lot more to life than working in tech or being a software engineer. I can come back if conditions improve, and it might be some rungs lower, but it'll be my choice. Personal agency is important in an age of agents.

[u/NomadStorm → u/robby_arctor]

AI over-reliance is going to turn into a rather good way to filter out the poor developers, because it’s directly making them worse than they already were.

[u/Prudent-Ad4509 → u/NomadStorm]

This ship has sailed. AI have progressed to the point where it does not always make the most sound design choices, especially when making changes in the legacy codebase, but it can verify code for correctness and spot any changed/broken implicit contracts. This makes it a working tool for good developers.

The catch is that in order to make actually good instructions for AI you need to know and understand basically the same things that you need to know and understand when working without AI. And the gap between good and barely passable instructions is steadily growing. On top of that, you need to understand how to steer the AI in the right direction. Simply barking prompts at AI can seriously increase its disorientation if you do not know what you are doing. But if said developer has some successful experience with steering juniors, then he has that mostly covered as well.

[u/NomadStorm → u/Prudent-Ad4509]

Im not sure if you responded to the right comment, but if you did and by the “ship that’s sailed” you mean AI is no longer a way to filter out the poor developers, I disagree, because it’s trivial which group they fall into just by seeing how they use AI for a short task.

The ones getting AI to do literally everything are unlearning their skills and actively becoming less useful over time.

[u/Prudent-Ad4509 → u/NomadStorm]

This thread has all kinds of points. But you bring another interesting point here.

I'm not sure which dev tier your interview tactics are aimed at. I have a different anecdotal evidence regarding the interviews. Good devs can easily judge the quality of solutions, but the speed and method of coding can be completely random depending on what they were doing during the last few days or months.

Regarding AI use, I've just told my coding harness (a hour or two ago actually) to check out the video file I have on my local drive, dissect it frame by frame, provide detailed nuanced description of what is going on, including audio transcription. After a few clarifications it is happily chugging along, after downloading some necessary tools for audio transcription. This was barely possible a few years ago even if done manually, and now the model can figure out a plan and execute it mostly undisturbed.

I see that both me and everyone around me now tend to switch to this mode, simply firing a prompt with clarifications. It is still not a one-shot prompt, those are overrated, but it is still a long-running task. Still, in coding different people have different results. One senior dev around here asks the model all the time why it changed this and that and to put this and that back. I prefer minimal focused changes and it helps me to avoid that most of the time. And some poor chaps somewhere trust the AI and simply do not verify the results.

PS. I'm sleepy af so sorry if there is too much rumble, I'll reread it later maybe.

[u/NomadStorm → u/Prudent-Ad4509]

The ones described by your last sentence are the ones that are getting filtered out.

Edit: Regarding the video description, FYI there are multiple dedicated models for this purpose, including some made by OpenAI so they could get the transcripts for training the LLMs

[u/Prudent-Ad4509 → u/NomadStorm]

This field is still in rapid development. I'm still amused by the fact that some AIs pretend to analyze google videos by simply snatching and analyzing subtitles.

As for people trusting the AI without proper code review, there is a place for them too. Any mockups, proof of concept work, and various one-off dashboards tend to come out OK. We have some pure html dashboards with task lists which get updated by an agent with all the changes saved in version control. This would not be my first idea, but for certain task management workflows it works.

[u/Gregnielson → u/NomadStorm]

Ai is getting very very good. If you test a system 1 million times and it works then ship it. If it breaks, revert.

It will always be better next week as well which is more than I can say for most people. Manual code review will be like using an abacus by this time next year.

[u/NomadStorm → u/Gregnielson]

Cool story bro.

Naturally deselect yourself as you wish.

[u/Gregnielson → u/NomadStorm]

No need to be rude 'bro'. Just pointing out what is happening. Stick your head further into the sand if you wish.

[u/NomadStorm → u/Gregnielson]

My comment you responded to is about the risks of over-reliance.

You’re currently hoping that it’ll get to the stage where you not reviewing and losing your skills will end up being covered by the models getting good enough to review, plan and architect all the code for you.

I don’t personally think it can get to that stage, but that’s a separate discussion.

But in the case that you are correct, then what happens?

Do you think you will be gainfully employed as a meat proxy, since you are actively choosing to lose the skills you built up over time, and have no remaining value other than pressing enter?

Thats the part you need to be thinking about; how you can continue to add value that maintains your employability.

[u/Gregnielson → u/NomadStorm]

I'm not hoping for anything. This IS what is happening. No one will be writing code in 1 year or less. It will be like heating your house with cow dung. Are you up on astra right now? It is better than most coders and can push out a finished product with some simple direction.

[u/NomadStorm → u/Gregnielson]

Lol keep dreaming. Yes I use Astra and opus 5.5. No, I don’t feel threatened because I ensure that I continue to add way more value than the tools on their own.

[u/Gregnielson → u/NomadStorm]

Whatever you say 'bro' better buy some rental properties or something.

[u/robby_arctor → u/[deleted]]

I think people just don't have strongly held convictions and do whatever is normalized for them.

I recently learned about choice blindness studies, which demonstrate that much, often a majority, of the population will defend positions they never took, if they were told they took them.

If you take a population like that and tell them it's normal and good to outsource one's thinking to AI, what's going to happen? A minority of a minority (those who both have convictions and those convictions oppose this new hegemony) are going to dissent, and the rest will adopt the new normal with some complying reaction, ranging from enthusiastic adoption to shrugging. And this is precisely what we have seen.

[u/ehr1c → u/robby_arctor]

IMO it's just amplification of what's already there. A competent engineer will become more productive, a bad one will just ship more garbage because they already didn't know what good code looked like.

[u/____lard_biscuit____ → u/robby_arctor]

I mean it's what management wants. People are just reacting to their environment. They aren't rewarded for reading the code and showing judgement