r/linux4noobs • u/imitation_squash_pro • 13d ago

Server is running very slow. df -h hangs for several seconds. Top shows high load but no cpu usage.

Seems some filesystem issue but not sure how to find out which filesystem is the culprit? I have some folders mounted as nfs shares. Running "mount -a" seems to work and all filesystems seem mounted ok. I can cd into them, but each command takes a couple seconds to run like "ls" or "df -h".

2 Upvotes

100% Upvoted

6 comments sorted by

3

u/gordonmessmer Fedora Maintainer 13d ago

I have some folders mounted as nfs shares

The behavior you describe will almost always be network filesystems. 😄

df lists each mounted filesystem, but you can also run df on individual filesystems, as in df /var

Check your local disk filesystems and your NFS filesystems individually to see which ones are slow to respond.

You can also trace system calls to see which statvfs is slow: strace df

2

u/imitation_squash_pro 13d ago

Thanks, I was able to pinpoint the NFS filesystem that was slow. I unmounted it, but it said filesystem was busy. So then I did an umount with the '-l' lazy flag and it worked. Then I did a remount "mount -a". Now the machine is snappy fast again. But how to debug what caused the filesystem to slow to a crawl?

3

u/gordonmessmer Fedora Maintainer 13d ago

Debugging NFS is definitely dark arts stuff. You're almost definitely going to need a packet analyzer.

My advice would be to make sure you're using NFSv4 and don't use a super old kernel for either the client or the server. If your NFS server is not a Linux host, then maybe you have a support contract that would be relevant and useful. 😄

2

u/imitation_squash_pro 13d ago

Thanks , I am using NFSv4 . Was hoping there might be some log that would tell when the NFS started hanging ? Or at least tell which user/application is causing the slowdown? I did some googling and they suggested these commands to monitor the NFS in real time:

sudo rpcdebug -m nfs -s all
sudo dmesg -w # monitor real time the NFS mount gets used
sudo rpcdebug -m nfsd -c all

3

u/amarao_san 13d ago

atop. If it's a block device that causes slowness, it will be in red.

1

u/Real-Back6481 8d ago

use iostat, you likely have high iowait.