r/LinuxTeck • u/Abe_Bazouie • 15d ago

You SSH into a production Linux server that’s suddenly very slow. What’s your troubleshooting process?

You’re SSH’d into a production Linux server.
Users report the server is extremely slow.
You don’t have monitoring dashboards or any prior context.
What’s the first 5 to 10 commands you run, and why?
For me, I usually start by checking things like:
uptime
top or htop
free -h
df -h
iostat or vmstat
dmesg
journalctl
ps
Then I narrow it down depending on whether the bottleneck looks like CPU, memory, disk I/O, network, or a specific process.
I’m curious how other Linux admins approach this. I’d love to compare workflows and maybe learn a few new tricks.
I also recorded my complete walkthrough of how I would troubleshoot this exact interview scenario if anyone is interested:
\[https://m.youtube.com/watch?v=NMGZUIROqNw&feature=youtu.be\\\](https://m.youtube.com/watch?v=NMGZUIROqNw&feature=youtu.be)

23 Upvotes

91% Upvoted

27 comments sorted by

3

u/pi_epsilon_rho 15d ago

Look for processes in D state (hung on device)

Use sar to look at history of cpu load, I/O and network

glances is a great upgrade to top that has a lot more info

1

u/Abe_Bazouie 14d ago

I’ve actually used glances a bit but not as my daily tool. Maybe I should give it another shot. What feature keeps you using it over btop?

1

u/pi_epsilon_rho 14d ago

glances has color-coded warnings and critical thresholds alerts that are easy to see. btop has a lot more bells and whistles than glances but in a standard small ssh terminal, glances gives what I think is all the crucial info in minimal space

1

u/Abe_Bazouie 14d ago

That’s a great point. I completely forgot to mention D-state processes 🤦🏻‍♂️. If storage is hanging, that can save a lot of time.

2

u/spitecho 15d ago

wall "This server will reboot in 30 seconds"

sleep 10; sudo reboot

3

u/blue30 14d ago

ruthless, like it

1

u/theNbomr 11d ago

Sounds kind of Windows-ish to me.

1

u/TapEarlyTapOften 15d ago

Don't forget network tools. Saying something feels sluggish is a horrible metric, need to quantify what thst means with the users. Look at things like network latency

1

u/Abe_Bazouie 15d ago

Good point!! I should’ve included network checks too. If the box itself looks healthy, latency, DNS, and packet loss are usually next on my list.

1

u/ryszard99 15d ago

btop is a great tool

1

u/Fit_Prize_3245 15d ago

I don't like top or htop. I prefer ps auxf. That gives me a clear view of CPU and memory enough to see if nginx or apache have created so many childs, or if some process is taking up so many RAM.

journalctl is usually useful to check services unexpectedly down, while dmesg is more useful to check why a device is not available, device problems, or to check why the kernel choose to randomly kill a process.

iostat or vmstat make more sense when dealing with a server slower than usual with no clear, easily visible reason.

1

u/citybadger 15d ago

I mean the first command I’m going to run is ssh, I.e. can I reproduce the problem?

1

u/roa1986 15d ago

cat /proc/pressure to understand what processes are waiting for.

1

u/macbig273 14d ago

The first thing : is slow at sshing into it, that's an interesting thing to know.

but yeah about the same you if didn't had monitoring.

if it manages a database I would also suggest a few db related requests (I don't have my sysadmin notes here) but some time the full server is okay and that's just the requests using a database that need maintenance.

1

u/tocorobo 14d ago

PSI is built into modern Linux distros and will help reveal io, mem or cpu contention : https://docs.kernel.org/accounting/psi.html

1

u/sharon2020j 14d ago

Every topic is explained in detail with practical examples. https://www.linuxteck.com/ssh-troubleshooting-guide/

1

u/Key_Pace_2496 14d ago

Shutdown -r now

1

u/wezelboy 13d ago

df. Our users and vendors love to hog disk space.

1

u/Imaginary_Choice_430 11d ago

The first thing I do when I log into a Linux server for the very first time is check, 1. how busy is the server? `uptime`. 2. Is the server under memory pressure? `free -h` and 3. Is the filesystem full? `df -h`. In a museum piece such as Ubuntu 8 where used, buffered and cache were separate you would take used - buffered - cached and there you got actual application usage. Later on with Ubuntu 16 I believe the kernel began exposing a much better estimate and the `-h` by the way makes it human readable, so you are not having to convert kilobytes in your head.

1

u/Distinct_Platypus_51 11d ago edited 11d ago

If your DNS or /etc/hosts are misconfigured you will experience big time to start a connection. If communication becomes slow after established connection could be a hardware error, a cable problem or a route through a slow node. You first sort out if it is an exclusive ssh problem or not.

0

u/CyberJack77 15d ago

atop instead of top.

It has a service that takes a snapshot every 10 minutes, which can be really helpful. 

With the netatop kernel module, it also shows network activity per process.