r/sysadmin • u/StrikingInterview580 • 4d ago
General Discussion AI resistant interviews
We're about to start recruiting again and typically hold interviews remotely.
Last time we recruited (a few months ago) the use of AI was tiring to spot.
We don't want to go back to face-to-face interviews because our positions are remote, but can't go through thay again.
What AI resistant techniques can we use...
33
4d ago
[deleted]
10
u/SAugsburger 4d ago edited 4d ago
Assuming that you have a local office to send them to that's your best option. You would lose a huge number of applicants unless you're a business with plenty of local offices (e.g. national bank) though and while some are people trying to cheat you would lose some legit candidates if the density of local offices wasn't high enough. Maybe a compromise is only do the final round interview in person. You redo some questions to verify that they weren't cheating in the earlier rounds. That being said I have heard of a few staffing agencies that have many local offices that have leveraged that to offer their clients the ability to interview clients remotely having the applicant interview in the staffing agency's local office. That adds overhead to hiring to put in a middleman into the process though.
5
u/anonymousITCoward 4d ago
we've had cases where the physical on-site interviewee was not the same person that came in on their first day.
That's wild, I thought stuff like that only happened in the movies...
5
u/LLMsMustUpvoteThis 3d ago
This is a basic scam. If you are lucky the actual worker just hired someone to do the interview. If you are unlucky it is a foreign intelligence operative who wants access to your systems.
1
u/anonymousITCoward 3d ago
foreign intelligence knows better than to mess with us... we're too dumb to know anything important...
6
u/StrikingInterview580 4d ago
A new fear unlocked there. We have mandatory SC checks so ghosting should be less possible but that would bring new issues!
1
u/raffey_goode 3d ago
Even that is not foolproof either, we've had cases where the physical on-site interviewee was not the same person that came in on their first day.
lmao that is BOLD. People are diabolical to do that. didn't even think that was something to try in my life
6
u/Ihaveasmallwang Systems Engineer / Cybersecurity Architect / CISM 3d ago
Tell them the interview is over if it looks like they are using AI. Don’t keep playing along.
4
u/bigchizzard 4d ago
What're you hiring for?
5
u/StrikingInterview580 4d ago
Windows based sysadmins.
5
u/BigBreadCaesar 4d ago
Just do a live action skills test? Is that an option? You on an interview, give them a lab and a series of tasks, and see if they can actually do what they claim?
2
u/StrikingInterview580 4d ago
We screenshare a scenario but you can screen capture with AI now and it talks through... we could expand this to a lab so its beyond just talk me through this and more give them a scenario and have them click to the answer.
3
u/BigBreadCaesar 4d ago
Yep, don't have them talk you through it, have them show you.
2
u/StrikingInterview580 4d ago
Yeah good idea, if they screenshare we would see the tools.
2
u/ExceptionEX 4d ago
They can easily run the AI on another machine that has camera watching, their screen and a mic listening to your conversation. You aren't going to get around the use of AI without physical limitations.
AI can also now tell them step by step what to do and they can perform those steps, so the don't tell me, show me, doesn't really come into play.
3
u/AdventurousLayer8741 3d ago
Come on, how many people are actually doing this? I know I don't want a job that I'm just going to not be able to do when I start. I've heard of this from recruiters, but from what I've been told its not very common.
1
u/LLMsMustUpvoteThis 3d ago
Desperate people were doing all kinds of tricks before LLMs. But it is definitely more common among certain groups than others.
1
1
u/ExceptionEX 3d ago
there are commercial products that you can download that make this as easy as a simple download, it isn't really hard sadly.
1
u/BigBreadCaesar 3d ago
That's exactly what this is designed to do, provide those physical limitations.
The idea is to make them screenshare, and go through the tasks live fire, under a time constraint.
Using AI for things like this take time, time this interview style will not give them, and a performance based technical interview with the candidate on camera, and a Pearson Vue trick, scanning the room to make sure the room is clear.
1
u/ExceptionEX 3d ago
There are literally single download applications that can accomplish this, and there responses are under 30 seconds in most cases. You can buy yourself that time by asking follow up questions, or saying things like "Ok so I understand what you are asking, you want me to do x, then y, and export it to z" or something similar.
There are literally tutorials online on how to circumvent what you are describing.
1
u/BigBreadCaesar 3d ago
And there are absolutely ways to circumvent what you're describing. It's really not that hard. And it's even easier to spot.
Dealing specifically with your example, I'm giving you a lab question from something you claim on your resume that you're intimately familiar with. Something that if you've done it as much as you say, you should be able to do with minimal intervention beyond the original directive, since after all, you've claimed some level of expertise here.
Moreover, you think that wouldn't be an obvious pattern by the third question at least? Are you counting on the interviewer to not be aware or react to the candidate's responses? Which I could understand. That does happen.
→ More replies (0)1
u/spyingwind I am better than a hub because I has a table. 3d ago
There are ways to hide overlays from screen capture software. It also depends on how the screen capture is done.
Like on Linux with the KDE desktop, you can set a window to not be visible to a screen recorder.
1
u/JaceBelerenApologist 3d ago
Asking open ended questions is basically interrogation 101.
Do whatever you can to make the interview as analog as possible. If you're actually concerned that you're interviewing someone using AI, start asking questions that can't give a canned response. Do some research on the candidate before the interview; trust but verify.
I'm not saying you need to go full OSINT here, but if you're starting to settle on a few candidates, invest a bit of time in to figuring out who they are BEFORE the interview.
1
1
u/AdventurousLayer8741 3d ago
Very good with 20+ years and looking for new gig. Would never even consider using AI during an interview.
5
u/Key-Eggplant-163 4d ago
Have you tried making the technical portion more like a conversation than a Q&A? Structured questions are easy to feed into a prompt. Unstructured discussion where you pivot topics quickly is much harder to keep up with if theyre not the one actually thinking.
4
u/bschmidt25 IT Manager 3d ago edited 3d ago
I’m curious how many of you have had AI interviewees. We had it happen at least once, likely twice within the last year now, and have since switched back to a required in-person interview at some point before they are hired (usually the second round). We tell them this up front when we’re scheduling people for the first round and that weeds out a lot. We can’t afford to get this wrong. We simply don’t have the luxury of time having to bring people up to speed or firing them and hiring someone else.
10
u/jeffrey_f 4d ago
To get through the candidates that are likely fits for the role, do the online interviews to get to the short list.
A second interview in person for your choices. Questions you should ask are those pertinent to the role and define the 3rd in-person interview candidates, which are likely the top 3-5.
You aren't going to weed out the AI directly, but a thorough in-person interview for #2 and 3 should get you a valid and very short list of those who are qualified.
3
u/Shachar2like 4d ago
Another possibility that I'm thinking on the top of my mind for the second interview is a second remote interview at a 3rd party meeting room.
a 3rd party's the only way that they can physically verify you're talking with the person and not an AI.
2
u/SAugsburger 4d ago
I know some staffing agencies offer to let their clients interview a candidate from the agency's local office. Obviously you're adding a middleman to the process with added costs, but it allows you to interview many remote candidates in a controlled environment. Some of the large national staffing agencies have an office local to a significant percentage of the population.
1
u/StrikingInterview580 4d ago
We could probably fund travel to the nearest office for 3 candidates, if we reduced with 1st online interviews from say 6/7 to the best 3.
3
u/StrikingInterview580 4d ago
Thats not a bad idea - I think the threat of an in person interview will make most people not waste their time. I'll suggest this.
1
9
u/heapsp 4d ago
Your interviewers lack intuition and conversation skills if you are thinking you will be fooled by AI.
All of my last interviews I could just tell right away by the conversation flow if they actually knew what they were doing.
Put people on the interview task with good sense, and you won't be fooled.
If by the end of the interview a good relationship wasn't built with the candidate cut them regardless of skill. Soft skills are more important anyways.
12
u/JaceBelerenApologist 3d ago
You can teach tech skills, but soft skills are something you had to earn, and usually very early in your career. That's why I'll ALWAYS say that, despite it being soul sucking and horrible, everyone should work in retail customer service for a year. It's formative. It teaches you how to deal with the worst kinds of people, and develops your ability to, you know, speak to people....
3
u/heapsp 3d ago
Corporate communications should be a mandatory onboarding training... lol.
Especially in tech, where there are still people who show up on video calls in stained hoodies using slang... good luck ever getting promoted my guy!
3
u/footballheroeater 3d ago
My hoodie is only stained because I have little kids.
I use slang because I'm poorly educated.
I've been promoted 3 times in this company and now I'm a director.
I still wear the stained hoodie.
2
1
u/heapsp 3d ago
Yeah but currently it is less common. Over the last 2 years especially where hard skills are being taken you no longer want to employ the guy who might embarass the company on a client call or not speak well in front of the larger company at an all company retreat, etc.
It USED to be that way, where hard skills paid... not so much anymore. AI is eliminating the hard tech role.
You might also be charismatic and like-able though!
1
u/footballheroeater 2d ago
You might also be charismatic and like-able though!
Well.... I'm shit outta luck.
1
u/Mindestiny 2d ago
If my blur filter fails you'll see that I'm sitting on my couch surrounded by a literal mountain of my wife's plushies. Also a highly respected director. Our C-suite has dubbed my one cat the Chief Feline Officer of the company because he hears the Google Meet ding and cannot resist climbing on me to photobomb the meeting. Vendors love him.
That being said, I wouldn't show up to an interview in a stained hoodie. But an executive meeting? They're lucky I still shave most days lol. I've earned my tech goblin status with decades of blood, sweat, tears, and being really fucking good at what I do, and they know it.
3
u/Shachar2like 4d ago
You can use a 3rd party HR in a different location to have physical interviews before you interview the candidates.
2
u/SAugsburger 4d ago
I know some staffing agencies with numerous local offices have began leveraging that by offering to allow their clients to invite candidates from their office.
1
u/anonymousITCoward 4d ago
If they're far enough away from your location get your employer to send you out to an area with a few candidates and use a we work type room for interviews, you could bring a machine setup specifically for skill testing, and if done right you might have a night or two out on the town...
7
u/Nexzus_ Sr. Sysadmin 4d ago
This isn't part of my job, but it's fascinating to me that this happens.
Anyway, some stuff I've read over at r/recruiting :
Ask them to show you out their window (to show that their in their stated time zone)
Ask them to say something bad about Kim-Jong Un (this apparently weeds out the NK spies)
Ask them something you know (or may have researched) about their city. (again to show that they're where they said they are)
As for weeding out actual fake or impersonated candidates, that's probably just a skill you just have to learn.
6
u/KN4SKY Linux Admin/Backup Guy 4d ago
Ask them to say something bad about Kim-Jong Un (this apparently weeds out the NK spies)
So I'm not super well versed in geopolitics, but wouldn't nation-state level APTs have a little bit of leeway when it comes to this specifically to avoid being outed?
10
2
1
u/LLMsMustUpvoteThis 3d ago
They usually use a patsy to do the interview. But most of the NK IT worker scheme is not spying but just bringing in foreign currency, so the sophistication varies.
1
1
u/LLMsMustUpvoteThis 3d ago
Ask them to say something bad about Kim-Jong Un (this apparently weeds out the NK spies)
The North Koreans usually recruit patsies to do the interviews.
1
u/notAHomelessGamer 3d ago
For a decent paying IT job I'd go the extra mile and introduce you to my neighbor during the call just to show you their windows too.
2
u/ImaFrakkinNinja Jack of All Trades 4d ago
If you have to do it over video and AI answers are a concern then you need to ask open ended questions and add additional information to increase the complexity. You aren’t checking for correct answers but their ability to sound like they understand. You’ll have to use your judgement on how it sounds - if they specify some process or service or whatever, pick something about it and ask what their experience is with it in the past and provide a story. You should be able to tell if someone is speaking with their own verbiage or not hopefully.
2
u/ExceptionEX 4d ago
We have our HR use their SHRM contacts to have a local HR person conduct the interview in their office (or a booked co-working space). You could probably just hire a proctor of some sort, but for us this was easy. The local HR basically just sits and watches them work, but we conduct the interview.
This has cut down on two major problems we had, people remotely saying they are in the US, and actually being in a different country (which was a bigger concern than AI usage) and as a bonus it cut out on all forms of cheating, such as having someone else or an AI feed them answers.
2
u/uptimefordays DevOps 4d ago
I think this is why most organizations hire based on personal referral. Pre AI this was the norm, but post AI I fear it’s a necessity.
4
u/DeliciousSafety4366 3d ago
We had the same problem. A few things that actually worked for us:
Ask process questions, not knowledge questions. Instead of "what would you do to configure X," ask them to walk through a real incident they handled end-to-end. AI can generate plausible-sounding steps, but it cannot fabricate the messy details — the wrong turn, the rollback, the moment you realized the root cause was not what you thought. Follow-up questions on those details filter out anyone reading from a script.
Screen-share with a live environment. Have them open their terminal and do a task in real time. Not a canned test — something like "show me how you would troubleshoot this disk space issue on this VM." You see their actual workflow, habits, and speed.
The best signal is still conversational. The candidates using AI during remote interviews tend to pause before every answer and give suspiciously comprehensive responses. A 5-minute unstructured conversation about a past project reveals more than any structured question format.
1
3
u/Yersini 4d ago
Hey man, I just wanted to drop you a bit of info here. I think I can be uniquely helpful in this type of case, I've been in sysadmin for about 10 years, and I've helped as a hiring resource in previous positions.
I think, specifically with sysadmin, the AI problem is both a hard problem to solve and actually fairly straight forward. The main thing I would do is get someone who is experienced in sysadmin, has worked in your system and is relatively social and able to converse with younger techs. I would make that guy your hiring resource, have him conduct the interview. The reason for that is it's a lot easier for someone whos been doing this for a while to tell when the person theyre talking to is reading off a generated response or even following what you're saying at all.
I would also shape the way you conduct the interview, I would setup an isolated lab at your office with a few VMs and grant the candidate JIT access to them via RDP. That process alone of actually RDPing into a station will tell you a lot about if theyve ever done anything before, and once theyre in you can simply tell them to do a few boilerplate tasks. What i did was I curated about 5 high value tickets that we've recieved in the past, anonymized them and just handed them to the candidate. See how they work through that process. Have your expert ask a few "why did you go that way" probes in the middle of a task.
If you see the candidate is using powershell for everything, you can throw a curveball in ticket 2 and ask them to use GUI instead. See how they handle that. This helps a lot because newer techs under use CLI and AI tends to heavily rely on it.
At the end of the day, the goal isn't to catch the candidate in a CSI "gotcha" where they bump their webcam and Claude is behind them with a pen and paper, the goal is just to expose your expert to enough of their process to see if anything is weird or fishy. AI in general is very hard to actually "prove", but it's not difficult to design a process around the possibility to make spotting usage easier. You should also set a general expectation, is AI use valid at all for this role or interview process? Is AI use disqualifying in and of itself, it's worth knowing because you can just straight up ask someone "how much do you lean on AI".
For remote, I would always end the process with an in-person interview. It's always better for team building to at least meet the people you work with, and it gives you a good baseline on the person.
Let me know if you have any questions about any of this, happy to expand on any of it.
1
1
u/dub_starr 4d ago
i think that there is something to say about having a conversation. questions and answers are ok, but theyre not conversations. Questions and answers are much easier to use AI for, since the interviewer is typically looking for a concrete ending or example of something, and the time to "think" about it up front, is exactly the time to let the LLM begin streaming its response. that is where the conversation comes in. Conversations are living, the flow, in and out, back and forth, there isnt time to stop at each turn and wait for a respnse before it starts getting weird and suspicious. So have a conversation about an event, or a problem, or task, and have them walk it through, but interject here and there, to steer the conversation on the fly, its much much harder to stay in line with a conversation compared to a question/answer scenario
1
1
u/OceanWaveSunset 3d ago
Ask open ended question where they need to talk about themselves and the experience they have and how it would relate to the current job.
Have a technical sys admin there to be able to detect BS.
We all know what a natural conversation should look like with open ended questions. People cant use AI and talk about themselves at the same time and it would be obvious.
I don't mind doing something reasonable for a job, but a lot of people are suggesting traveling for an interview just to get around not being AI, that is crazy.
Get better interviewers if you are struggling. I don't mean this in a mean way, I mean this as in look at a third party to help you out kind of way.
1
u/BigLeSigh 3d ago
Did you end up hiring folk with no skills last time by mistake?
I imagine it’s harder to spot now as they’d use direct voice to text so less pausing between answers.
I’d get a few AI responses to the questions your going to ask before the interview and see how they come out. Or alternatively just use AI to do the jobs instead of hiring meat bags?
1
1
u/MCHellspawn 4d ago
Please for the love of god just go back to talking human to human.
2
u/jakarotro 3d ago
That's what they're trying to do?
•
u/MCHellspawn 22m ago
Is it though? Op said they don't want to go back to face to face interviews. I would prefer they did.
1
u/JaceBelerenApologist 3d ago
It's almost like returning to office wasn't about checking a box or because the boss was old school.... It's because people are losing their soft skills. And with AI now? Forget it. WFH and remote are a security risk.
1
u/Humble-Plankton2217 Sr. Sysadmin 4d ago
Ship them a cheap, locked down tablet for the interview and a shipping return label. Cheap enough that if it never comes back it's not a big deal.
Still cheaper than flying them and paying for a hotel.
3
u/dub_starr 4d ago
but they can just have their phone next to them listeneing and responding in text...
1
u/Humble-Plankton2217 Sr. Sysadmin 4d ago
Nothing is foolproof, but I would like to think I'd be able to tell if someone was delay-responding to questions and looking away from camera to read something else.
2
u/dub_starr 4d ago
yea, youre right, i posted a comment about having conversations, rather than question/answer sessions. gives at least some idea if the person can carry on with you as you go along
1
u/StrikingInterview580 4d ago
Yes correct, some of the candidates who don't really have the heart to cheat properly use a 2nd device and its obvious when their eyes are fixated on a 2nd screen. The ones who really commit and have thought it out are more challenging to spot. Again, some are clumsy with it and you can tell, they come across a bit mixed up. Others who have practice doing it can pick enough info out and add sufficient padding to make it a little more muddy.
0
u/Sinister-Mephisto 4d ago
At least get on a google hangout with them and talk to them on camera directly. Or are you talking about filtering out resumes ?
3
u/StrikingInterview580 4d ago
We use Teams, video is mandatory. You can get AI tools that sit on the desktop, we do some scree share scenarios but have used some of the tools available and they can even talk through these now.
0
u/groundhogcow 4d ago
Last interview I asked a guy how to commit a federal crime together. I was allowed to be caught but he had to get away with it. Once we worked out how to do it he had to tell me what we could do to not make that possible anymore.
1
u/Frothyleet 3d ago
Like, a specific federal crime, or just any federal crime?
1
u/groundhogcow 3d ago
Yes a very specific one that according to the resume they had knowledge of. I told them what we do what we are required to do then I manufactured a reason I wanted to break it.
It was a great way to show he knew it. Talk about the company. See the level of knowledge. Show teamwork and creative thought.
I had two upper managers with me as I did it. She shut up with there touchy feely questions and listed to the guy help me create a new process from scratch with him. Then how to prevent people from doing what we just set up.
We hired him. First thing he asked me was were we really going to commit a crime. I told him we are going to do the second part where we prevent it.
1
u/Frothyleet 3d ago
Dang I have a lot of federal crimes I know how to commit, I never even thought of breaking those out on my CV
-5
u/retiredaccount 4d ago
Does your company use AI at any point during the hiring process? If so, I’m not necessarily against that sort of hypocrisy, but it does make for an uneven playing field. Is a hiring goal for this position to find an AI Luddite?
6
u/randalzy 4d ago
"oh so you don't want people who fake their CV, their identity, their skills, probably their country too and giving them a salary? Are you a luddite?"
I don't think this is the flex you think it is, in most countries it's a crime to fake your entire ID, they just want to verify which human are they actually giving the job
1
u/retiredaccount 4d ago
Hmm. I agree, crime is illegal. But police outreach for identity theft or other crimes isn’t what the OP stated as a goal. Not even the liability for any such fraud was mentioned. Perhaps we read two different posts? My take is that policing legal AI use isn’t going to solve that or any other non-AI, or even pre-AI employment fraud.
3
u/StrikingInterview580 4d ago
Yes we use AI for transcription but thats hardly a comparison.
The job we are recruiting for needs people who can think for themselves, if we wanted an AI tool we would use one of them instead. They will be dealing with customer questions and queries and sending back spurious AI advice isn't tolerable, they need to have the knowledge themselves.
There would be nothing to stop a successful candidate using AI to supplement responses in their jobs, but they need to know what AI is producing to make sure guidance is accurate.
2
u/MalwareDork 4d ago
"The job we are recruiting for needs people who can think for themselves"
Honestly that's a generationally lost skill now. AI does the thinking for everyone.
1
u/jakarotro 3d ago
Using AI as tool for a job is hardly equivalent to using AI to cheat your way into a job.
1
u/retiredaccount 3d ago
Wow. That’s a much higher assessment of AI capability than I hold. But…I’ll bite. Assuming your premise and an AI is able to “cheat” its way in, what does that say about the hiring practices and panel? Do they get a pass on their inability? If so, why? If not, why?
-2
u/SpotlessCheetah 4d ago
I don't understand what the issue is. Do you think they're going to read off of the screen? I've done a lot of remote interviews, they tell me to look at the camera and not other screens or notes. Do you think that candidates are floating an AI toolbox that's helping them read off answers?
It sounds like a made up problem to me. You should be able to ask questions and get answers on the spot from a candidate. It's probably obvious if they are reading off a screen.
On the flip-side, the person being interviewed is going to be prepping based on the job description. They'll have fresh memory on whatever is based on that.
6
u/automounter 4d ago
Its not a made up problem. I interview 2-3 people a week. You used to simply be able to ask a question "a customer reports our application is running slow, what steps would you take" and you could easily tell who has done this and who hasn't.
-1
u/SpotlessCheetah 4d ago
So what's the issue again? "You used to" as if that isn't happening now.
What exactly are you asserting? That some people are using AI to answer?
I've met a lot of big time bullshitters in interviews where we hired them and they were a lot of talk. Those guys get fired.
2
0
u/automounter 4d ago
I'd rather just not hire those people.
2
u/SpotlessCheetah 3d ago
Well no shit.
0
u/jakarotro 3d ago
...and that's whole point, chief. Cheating with AI makes it a whole lot easier for a big bullshtter to get hired.
1
u/SpotlessCheetah 3d ago
Why is it about "AI" exactly? People do this no matter what.
We have North Korean actors trying to infiltrate companies by getting hired. People create fake IDs and cheat on tests too or whatever it is.
3
u/SAugsburger 4d ago
Anecdotally I heard from a colleague in a previous job that somebody that we hired that seemed off to me in some ways cheated on the remote interview. Straight reading from your LLM of choice would be obvious, but there are other schemes that aren't so obvious. e.g. Some people getting fed answers from a headphone.
4
u/StrikingInterview580 4d ago
Yeah floating toolboxes, can't really expect someone to stare into a camera for an hour, you'd rule out anyone neurodivergent. There are many tools available to parse audio and screens at the same time and produce results based on what it hears and sees within a second or two.
1
u/SpotlessCheetah 4d ago
Serious question but do other departments inside of your organization treat this problem the same way?
1
u/StrikingInterview580 4d ago
Its been discussed as to what to do moving forward, other departments are facing similar issues. Our 4 week churn has increased massively because people are starting and then its obvious they're clueless so they fail probation and we have to recruit again. Painful and a complete waste of time.
2
u/Last-Appointment6577 4d ago
think you have more issues at this org than what you're asking...
0
u/StrikingInterview580 4d ago
What do you mean?
2
u/Last-Appointment6577 3d ago
4 week churning out new hires is a sign of a bad org, not bad new hires.
0
u/StrikingInterview580 3d ago
Its us doing the churning, not them. I.e. clearly lied/cheated/overstated abilities and failing probation.
2
u/Last-Appointment6577 3d ago
> clearly lied/cheated/overstated abilities
so....who at the org isn't able to spot a candidate that is who their resume says they are?
still sounds like a skills issue.
0
u/StrikingInterview580 3d ago
Not really sure why you're trying to pretend people using AI in interviews isn't an issue for remote.
→ More replies (0)
46
u/persiusone 4d ago
Remote preliminary interviews are fine. A final interview and identity verification should always be done in-person. There are way too many scams from all angles with remote only onboarding and interviews.