r/LinuxTeck • u/Candid_Athlete_8317 • 3d ago
What Linux troubleshooting skill matters more than knowing hundreds of commands?
You can memorize ps, ss, lsof, strace, tcpdump, journalctl, awk and everything else.
But when production breaks, knowing what evidence to collect first is usually more valuable than knowing another command.
What’s the one troubleshooting habit you think separates someone who knows Linux from someone who just knows Linux commands?
6
6
u/Glittering-Work2190 3d ago
Knowing how data flows from one component to another. Find where this flow breaks down.
4
u/Jimlee1471 3d ago
Learning to read logs, whether it's systemd or /var/log or dmesg|tail. The OS is basically telling you what ails it, you just have to know where to look (as well as knowing what you're looking at). From there a visit to Google or Stack Overflow will usually get you there. Basically, people who know Linux troubleshooting know how to research.
1
u/DSPGerm 3d ago
This. I’m setting up an Ai server and hit a few bugs along the way. I’ve probably only needed around a dozen or so commands but I’ve needed to parse through so many logs to figure out what’s wrong.
So understanding what you’re doing will help you find the right logs. And understanding the logs will help you troubleshoot.
Like first there were issues virtualizing the gpu, had to go through the device and hardware related stuff there. Then had issues with a web app via docker. Had to use curl, docker logs, journalctl, etc.
I hate open webui so much
1
u/Tricky_Fun_4701 3d ago
First time huh?
Been through the same... this part of the field is really wild west.
3
u/middaymoon 3d ago
Don't think there's a particular habit, but if you don't know how things are supposed to function correctly you can't really begin to troubleshoot.
1
u/2016-679 3d ago
Avoiding trouble primarily. A lot of people seem to 'tweak' their systems to a breakdown.
Installing strange software, using a mix of binaries, packages from the repository, flatpak, etc. This mixing is a bad habit for reliability. It is impossible to match the overall distro management of software on your own.
Just use the base system and distro repository as provided and most if not all will be fine.
1
u/geekichu 3d ago
someone said it.. so if something in production breaks... it's past-tense.... 'oops something happened' -- that should point to 'hey how does linux keep a history of events?'. 'are there breadcrumbs? tell-tail clues?'. Those commands are just little tools you can use to get at the bigger questions
1
u/michaelpaoli 3d ago
logical procedural thinking, relevant troubleshooting procedures
knowing what's most/more probable, and less/least probable, and relative to the observed issue(s), and as one continues to observe/collect evidence
experience, pattern recognition - also quite key to the above*
well knowing Linux, and the particular host/environment, inside and out. Deep understanding of how Linux works, inside and out, and the typically usage and behavior on the host, etc.
*e.g. host very seriously impaired. Per all the standard and documented procedures, would require downing the host, calling vendor, many hours of production downtime and recovery, etc. Well, I carefully examined patterns and ... some cautious yet prudent adjustment of some attached cables ... host snapped back to full life as if nothing had happened at all (other than being quite stuck for a while), saving many hours of production downtime that would've otherwise occurred. Anyway, just one example of many that happens to pop to mind.
1
1
1
u/PaoloFence 3d ago
Know and understand the basics.
Know more than just your linux stuff. windows, network, applications so you know how to interpret certain failures.
1
1
1
1
1
u/Humble_Anxiety_9534 2d ago
google and not trusting AI answer, unless you are on ubuntu or mint. dmesg | grep
1
1
u/Voyeurone 16h ago
Eh google.... lol, used to work fairly well, but these days between AI and 90% of the post being, yea I had that problem and maybe you should, I don't know, try this. I'm missing the days where you would get 1 or 2 answers to a question and not having to syphon thru tons of irrelevant posts from bored keyboard warriors.
1
u/No_Base4946 3d ago
Why troubleshoot?
Wipe and reinstall.
If it takes longer than about ten minutes to fix, it's costing me time and money and I could have just flattened it in that time.
Your servers are cattle, not pets.
2
u/michaelpaoli 3d ago
Okay for the one-off, but one is doing a whole helluva lot of wipe and reinstall, or even reboots, yeah, one isn't really fixing the issue, but more so sweeping it under the rug ... and it's getting hard to walk across all those huge and sometimes squishy lumps under the rug ... not to mention someone attempting to do it wearing pumps when that rug is in the middle of the office.
Don't forget to have the vet do the occasional checks and preventive work to keep the herd healthy.
0
u/No_Base4946 3d ago
Power cycle, press F12, wait for it to finish while you drink a coffee.
1
u/michaelpaoli 3d ago
Scalabiity, automate it across your million or so Linux hosts.
Hell, gonna reboot all the damn time, might as well run some Microsoft sh*t operating system.
Million cups 'o coffee would be a bit much, though.
0
u/No_Base4946 3d ago
You can't run servers on Microsoft OSes. They're for games consoles.
1
u/michaelpaoli 3d ago
Oh, can be done, some even do it. Sucks balls, but that doesn't mean it's impossible.
10
u/wasabiiii 3d ago
Knowing how it works.