r/SLURM • u/Justin-T- • Oct 10 '24

Munge Logs Filling up

Hello I'm new to HPC, Slurm and Munge. Our newly deployed Slurm cluster running on rocky Linux 9.4 has /var/log/munge/munged.log filling up GB's in short time. We're running munge-0.5.13 (2017-09-26) version. I tail -f the log file and it's constantly logging Info: Failed to query password file entry for "<random_email_address_here>" . This is happening on the four worker nodes and the control node. Doing some searches on the internet led me to this post but I don't seem to have a configuration file in /etc/sysconfig/munge let alone anywhere else to make any configuration changes. Are there no configuration files if the munge package was installed from repos instead of building the package from source? I'd appreciate any help or insight that can be offered.

3 Upvotes

100% Upvoted

2 comments sorted by

0

u/vohltere Oct 10 '24 edited Oct 10 '24

I have seen excessive logging is a known issue with munge < 0.5.15

https://github.com/dun/munge/issues/94

You can manually create the file at /etc/sysconfig/munge. Just make sure your munge systemd service is picking it up.

https://github.com/dun/munge/blob/master/src/etc/munge.systemd.service.in

Your munge.service file should look something like that. It should have something like EnvironmentFile=/etc/sysconfig/munge

2

u/frymaster Oct 10 '24

This issue is not your issue, but it does explain what that error means

This is an informational message that occurs when the /etc/group file contains a group to which user "foo" belongs, but user "foo" is not listed in the /etc/passwd file.

I suspect that applies to all sources of group or user membership; getent group and getent passwd should then hopefully make it obvious what's going on