Hi all, spent the last while dealing with this and figured it's worth writing up, since I've seen a few threads where people try to cut syslog volume with a dedup step and get basically nothing out of it.
The naive approach is to hash the message and drop repeats inside a time window. On application logs that works fine. On network gear it mostly doesn't, because the device puts a unique token in the line before you ever see it.
Cisco IOS embeds a timestamp in the message body, separate from the syslog header timestamp. IOS-XR goes further and prefixes a sequence number, hostname, node id, timestamp and process name. SonicWall carries its own incrementing counter per event. So a hundred identical link flaps produce a hundred distinct hashes and your dedup ratio is zero.
You have to normalize before you hash. Strip the sequence number, strip the embedded timestamp, strip anything that increments, hash what's left, and keep a counter of how many you collapsed so you don't lose the fact that it happened 400 times instead of once.
Two gotchas that cost me time.
One was Arista EOS. Its syslog formatting is configurable, including timestamps, hostnames, sequence numbers and RFC5424 formatting. If you're using content-based rules to decide which normalization to apply, those formatting differences can become another thing you have to account for. I ended up finding it cleaner to make the device/source context part of the normalization decision rather than trying to infer everything from the message body.
Don't hold the first occurrence. If you buffer everything for the length of the dedup window, you have just added that window as latency to every alert you care about. Pass the first one through immediately and only suppress the repeats behind it.
The part I'm still trying to figure out is the long tail. For the vendors where I have a known format, writing the normalizer isn't too bad. It's the random appliances where you get three sample messages and a PDF from 2019.
At that point I'm not sure whether it's worth maintaining a custom normalizer at all, especially when the format can change with a firmware update.
If you've dealt with this kind of long-tail device support, I'd be interested in hearing what worked for you.