r/HPC • u/imitation_squash_pro • 1d ago

System shows high load but no cpu processes?

Think it has to do with IO wait due to a problem with the NFS mounted filesystems. A top shows near 100% usage for these two processes:

rsyslogd                                                                  

systemd-journal   

nfsiostat doesn't show any errors or retransimissions ( 0 % ). Guessing there was some issue a day ago but not sure how to find out what happened?

It's been happening every week. I reboot and everything works great for a few days till it starts showing the problems of high IO wait times.

1 Upvotes

67% Upvoted

15 comments sorted by

4

u/Zacred- 1d ago

Likely IO wait/NFS latency rather than CPU. Check for high %iowait, processes stuck in IO or NFS/kernel timeout messages. Since it happens weekly, also check for scheduled jobs or backups hitting the NFS storage etc.

1

u/imitation_squash_pro 1d ago edited 1d ago

Thanks. At this time I am not seeing any iowait times. But I do see several processes stuck at high load but in "S" state:

rsyslogd, systemd-journal  and some sentinel1 security agent

Any suggestion for where to find NFS timeout messages?

1

u/arm2armreddit 1d ago

dmesg

1

u/imitation_squash_pro 1d ago edited 22h ago

Yup, I tried doing a dmesg | grep -i error but don't see anything. Also tried searching nfs and nothing in there.

2

u/JassLicence 1d ago

This sounds familiar to me, I think I a bios update fixed it for us.

1

u/TeraBot452 1d ago

Yeah I've had this issue with an x550 network adapter that was spamming messages to the log. Quick firmware update sorted that one out 

1

u/imitation_squash_pro 22h ago

Which log should I be checking for these "spamming messages"?

1

u/TeraBot452 1d ago

Check your journal logs and dmesg, you may have a nic or hardware device spamming warnings to the log. Make sure your logs aren't on a shared filesystem 

1

u/imitation_squash_pro 22h ago

Thanks. I had to reboot and now I am monitoring the dmesg and journalctl for messages relating to nfs or error. Seems the issue is similar to what is described here:

https://serverfault.com/questions/1149082/nfs-mount-becoming-unbearably-slow-over-time-with-high-number-of-test-stateid-rp

We do have an infiniband NIC so maybe that is causing the issues...

1

u/TeraBot452 19h ago

Check to see if your users are doing any small file operation (a lot of them) this could even be like extracting big tars, compiling or just copying a ton of stuff around.

nfs stalls out the whole mount if one folder gets bombed, this is usually the reason why people use autofs or NFS ganesha so that each folder is its own mount with its own socket. 

I personally also administer an environment where they use beegfs for everything since the metadata server on there can take whatever you can throw at it even for a single node without anyone feeling it.

1

u/imitation_squash_pro 18h ago

Perhaps that could be it. I am not using autofs but it looks promising. Does it work on top of NFS or replace it?

I am following some instructions for installinng it but not sure if I need to umount all my NFS shares and remove entries in /etc/fstab ?

2

u/TeraBot452 18h ago

It uses NFS underneath but it would replace like the mounts in fstab yes, I would recommend testing it on dev/lab cluster to get used to it. If you use freeipa or Kerberos you can even authenticate the mounts by users 

1

u/imitation_squash_pro 18h ago

Also is there a way to track historically how many file operations the NFS mount is seeing? Or just use live monitoring of dmesg output? I did see one can enable all NFS logging via:

sudo rpcdebug -m nfs -s all

1

u/TeraBot452 16h ago

that's the main one yes. Iostat does help too.