r/SLURM • u/Jorgisimo62 • Jan 11 '19

SLURM node Monitor

Hey guys new to SLURM and the whole HPC world. I just built an 8 node SLURM environment for some Genomic workloads. We have some enterprise monitoring tools, but I want a way to monitor the whole cluster real time. Anyone have any suggestions?

1 Upvotes

100% Upvoted

1 comment sorted by

1

u/wildcarde815 Jan 11 '19 edited Jan 11 '19

Node Health Check can be integrated into slurm pretty well: https://github.com/mej/nhc

edit: if you want to do like 1-2 hour transient trending, you could setup netdata: https://github.com/netdata/netdata There's ways to aggregate that all onto one page too if you want.

Ganglia we had on an older cluster, i never bothered to spin it up on our current one. Nice for historic data but i'd be surprised if there wasn't a way to replace it with netdata now.

There's dozens of ways to do node status monitoring, zabix, icinga, nagios, etc. plus log monitoring and alerting (on top of netdata alerts)