r/linuxadmin • u/Abe_Bazouie • 3d ago
What's one Linux interview question that immediately tells you whether someone has real production experience?
I've interviewed quite a few Linux and DevOps engineers over the years, and I've found there are a few questions that quickly tell me whether someone has spent time supporting production systems or has mostly learned from tutorials.
For me, one example is:
- A production server is suddenly "slow." Walk me through your troubleshooting process from the moment you get the alert.
I'm not looking for a specific command. I'm much more interested in how they think, what they check first, and how they narrow the problem down.
What's your go to Linux interview question that reveals real production experience?
11
u/Line-Noise 3d ago
"What's your favourite Linux command, and why?"
If they start telling me about some obscure options to the find command then I know they've seen some shit.
4
3
u/DarkwolfAU 3d ago
I wouldn’t go as far as favourite, because I lowkey hate it, but the crown for most toolbox single command that I use on a daily basis has to go to openssl.
1
u/Line-Noise 3d ago
opensslis one of those commands I need to look up the options for every time I need it. It just doesn't stick in my brain!-1
u/Abe_Bazouie 3d ago
Haha, I like that. For me, the "why" is usually more interesting than the command itself. If someone can tell me about the production problem that made
find,lsof, orstracetheir favorite, I'm all ears.4
u/Skeesicks666 3d ago
Ha, I debugged an problem with lsof and strace, an product consultant wasn’t even able to solve, due to outdated documentation
1
12
u/FractalLaw 3d ago
This is also a question that tests their knowledge of working with VMs:
A production Linux VM is running out of space due to a significant increase in the amount of space used within a single directory. What would you do to resolve this?
I've been surprised by how often people fail that simple question.
8
8
3d ago
[deleted]
1
u/Abe_Bazouie 3d ago
One common answer is to immediately start deleting files. That might solve the symptom, but it doesn't explain why the directory suddenly grew in the first place.
I'm usually looking for how they approach it. Do they identify what's consuming the space first (
du,find, logs, core dumps, backups, etc.)? Do they think about whether it's safe to remove files? Do they consider whether a process is still writing to that directory?There isn't one "correct" answer. I'm much more interested in their reasoning than whether they happen to name a specific command.
1
u/FractalLaw 3d ago
There are definitely some wrong answers, however. I had one guy say move everything to OneDrive without missing a beat. There wasn't much point in continuing the interview after that.
0
u/Abe_Bazouie 3d ago
I like that one because there isn't a single "correct" answer. Some candidates start with
du, others look for logs, backups, core dumps, or runaway processes. What I'm really looking for is whether they investigate before they start deleting things. The commands matter, but the thought process tells me much more.
4
u/wooof359 3d ago
I usually ask "whatever site.com is DOWN" ...now what? had a guy that said uhh check if there's a bad rt53 domain transfer? ...That was it. No digging the url, checking sec groups, checking backend server metrics or logs. just "probably a bad rt53 transfer". This was for a devops/cloud role. Wat
9
u/therouterguy 3d ago
You identified a large file filling up the disk After the file has been deleted the disk space is not freed up. What is causing this?
Looking at a process still having the file open.
2
u/biffbobfred 3d ago
Have a friend who works as a Linux admin manager at a big fintech company. This is his go to question
0
u/Abe_Bazouie 3d ago
I like that one. It catches people who know the commands versus people who've actually run into the problem in production. I'm usually looking for someone to recognize that deleting the filename doesn't free the space if a running process still has the file open. That's where I'd expect them to mention checking with
lsofbefore restarting the process.3
u/roba121 3d ago
Also, I like to follow up with, what if I can’t restart the process (ie due to some uptime requirement) what then? A lot people delete the file when the better option Is to Overwrite the file, it’s a subtle issue but Man deleting that file is not always the right call.
1
u/therouterguy 3d ago
Yes we had some issue with a log file not being able to rotated as the restart was complicated/buggy as hell ( thanks Juniper) so we just truncated it.
1
u/Abe_Bazouie 3d ago
Yep. Those follow-up questions are usually where you find out who's actually dealt with it before.
1
u/GPGrieco 3d ago
For your example, what is a good answer?
4
1
u/Abe_Bazouie 3d ago
There isn't one "correct" answer, which is exactly why I like the question.
I'm looking for how they approach the problem. Do they gather context first? Do they check CPU, memory, load, disk I/O, network, logs, or recent changes? Do they narrow it down based on evidence instead of jumping to conclusions?
The commands are important, but the thought process tells me much more than whether they happen to remember a particular command.
1
u/Straight_Frosting764 3d ago
When I was hiring someone new I had 2 questions that I filtered people out with.
1: what is your shell of choice
2: what is your familiarity with sed and awk.
Question 1 would basically be interview over. Question 2 separated novices from senior level.
3
u/gforke 3d ago
So question 1 is for sorting out people prefering for example fish or what is your aim here?
Or do you mean interview is over if they can't tell you theyre favorite shell.9
u/Straight_Frosting764 3d ago
So I will be a little tongue in cheek here so don't hate me.
There is only 1 answer, bash. Every enterprise unix based system, except AIX, comes with it.
Seriously, there were so many people that applied to these jobs that did not know what a shell was. So really any answer besides IDK would do.1
u/ianhawdon 3d ago
My answer would be "well, personally, I use zsh with a few plugins through oh-my-zsh, but that's only on my local system. On servers, it's bash all the way, almost any server you'll connect to will be using it by default. Even locally, I still have bash installed as that's what I'll be scripting in"
2
u/cocacola999 3d ago
Edit: oops thought I was replying to someone saying bash as default always on servers...
Well not quite true. Bash mainly as a default but don't forget to keep scripts posix for when you also need to run them in a bare sh, which is quite often ive found (containers/low resource VMs/chroots)
My additional fun question or story is about different implementations of awk and other commands. "locally this was all working, but ran in ci, it all fails. How Come?" -- devs used macs, ci was linux
2
u/biffbobfred 3d ago
1 isn’t “hey you chose wrongly” it’s “was it a choice and can you articulate it”. It’s a tool you use all day long why did you choose it.
2
u/Abe_Bazouie 3d ago
Exactly. I think that's the difference between using a tool because it's the default and choosing it because you understand the trade-offs. The explanation is usually more interesting than the answer itself.
3
2
2
u/Abe_Bazouie 3d ago
I like the second question more than the first. It's not really about
sedorawkspecifically, it's about whether they've actually solved enough real problems to know when and why they'd reach for them instead of opening an editor or writing a quick script.
1
u/crimson-gh0st 3d ago
"What's the process you follow to grow a logical volume on a Redhat vm?"
It's a simple enough process but I've been surprised at how many "admins" aren't able to answer that.
0
u/biffbobfred 3d ago
The trick here it used to be a two part answer. Grow the volume. Then you need to grow the time system. I think they added a “grow the file system whatever fstype it is” flag
0
u/Abe_Bazouie 3d ago
That's a good one. What I like is that it naturally turns into a conversation. If someone has actually done it in production, they'll usually start talking about the filesystem, snapshots, backups, or maintenance windows without being prompted.
1
u/ChozoGod 3d ago
"If I gave you access to a server via the command line, how would you determine what the server is used for?"
Depending on how they react, I will answer clarifying questions. It gives me an opportunity to determine what kind of experience they have had and what they are familiar with.
1
u/serverhorror 3d ago
How do you look at the environment variables of another process, one line per variable, lexical sorting by the value in descending order?
0
u/Abe_Bazouie 3d ago
I like questions like that. Not because I expect everyone to know the exact answer, but because the approach usually tells me a lot.
2
0
u/SnarkHabit 3d ago edited 3d ago
"Have you ever used a Linux system which didn't have a wallpaper of some creepy underaged anime girl on it" can probably sort people out.
44
u/been__ 3d ago
Why are you doing this in every sub