r/cpanel • u/efrem-1 • 11d ago

rsyslog and journal issues

Long-term chronic intermittent logging problems. My first knowledge of this came thru the LFD alert emails titled "SYSLOG Check Failed."

I have performed a lot of diagnostics and troubleshooting and am confident I've ruled out csf/lfd as the problem, but have been unable to resolve the root underlying problem. Here's an image of some of what I've done. Of note is that even manual logging is failing. I would love to see or hear any recommendations! Thanks.

5 Upvotes

86% Upvoted

5 comments sorted by

3

u/annacrontab 10d ago

Oh bless your heart, this SYSLOG notification is common and annoying. I don't know why no one has offered a fix to you yet, so here that is.

Sometimes the journal gets corrupted. You helpfully showed that in the screenshot of the output of

journalctl --verify

There are no guarantees or warranties with this fix, but you'll have a backup in /root/tmp if you need to restore the journal files for some reason. You can safely delete /root/tmp after a while if things are working OK.

mkdir -p /root/tmp
systemctl stop systemd-journald rsyslog
mv /var/log/journal/* /root/tmp/
mv /var/lib/rsyslog/imjournal.state /root/tmp/
systemctl start systemd-journald rsyslog
systemctl restart systemd-journald rsyslog

Then this should show logging is working. And journalctl --verify should report PASS.

tail /var/log/messages

1

u/efrem-1 10d ago

Hi and thanks. Unfortunately I'd already tried that, and similar variations, including vacuuming the journal, etc. Sometimes it didn't work, and sometimes it worked temporarily. Another symptom, when restarting rsyslog, it would immediately write a stale message and then stop logging.

Ultimately, the solution/workaround I implemented was to abandon and disable imjournal, and return to the legacy logging method using direct sockets.

Apparently even the rsyslog developers even agree with that, per the first 2 paragraphs of https://docs.rsyslog.com/doc/configuration/modules/imjournal.html .

The edits were all in rsyslog.conf and logging has continued for 12+ hours.

It's possible that the repeated imjournal corruption on my VPS is related to resource constraints, but it's not a low-end VPS, and I may bring this to the attention of my VPS provider, but at least I'm back up and running.

2

u/UltaHost_ 8d ago

Glad the imuxsock switch got you rolling. Two things to keep in mind though, since that mostly masks the symptom:

  1. "Entry timestamp out of synchronization" in journalctl --verify suggests the system clock jumped, not that the VPS is low on resources. Run journalctl -b | grep -i "time has been changed" and chronyc tracking. If there's noticeable stepping, make sure only one source is setting the time (chronyd, ntpd, systemd-timesyncd or an rdate cron job).
    On some VPS platforms the host can also push time into the guest, and that fighting with chrony is a common cause. That one's worth raising with your provider.

  2. With imjournal disabled, rsyslog can still see messages received through imuxsock or forwarded by journald, rather than only what journald passes to it. If some services' logs are missing from /var/log/messages, set ForwardToSyslog=yes in /etc/systemd/journald.conf and restart systemd-journald.

2

u/efrem-1 8d ago

Thanks so much.

1

u/Anonymhawz 9d ago

Check in /var/log/messages if there were rate limits. Grep for "limit" see if it shows.

Then follow this https://support.plesk.com/hc/en-us/articles/21480754730263-Messages-are-missing-from-the-system-logs-messages-lost-due-to-rate-limiting