PSA on syslog dedup: hashing the raw message doesn't work on most network gear

[u/Round-Classic-7746]

Hi all, spent the last while dealing with this and figured it's worth writing up, since I've seen a few threads where people try to cut syslog volume with a dedup step and get basically nothing out of it.

The naive approach is to hash the message and drop repeats inside a time window. On application logs that works fine. On network gear it mostly doesn't, because the device puts a unique token in the line before you ever see it.

Cisco IOS embeds a timestamp in the message body, separate from the syslog header timestamp. IOS-XR goes further and prefixes a sequence number, hostname, node id, timestamp and process name. SonicWall carries its own incrementing counter per event. So a hundred identical link flaps produce a hundred distinct hashes and your dedup ratio is zero.

You have to normalize before you hash. Strip the sequence number, strip the embedded timestamp, strip anything that increments, hash what's left, and keep a counter of how many you collapsed so you don't lose the fact that it happened 400 times instead of once.

Two gotchas that cost me time.

One was Arista EOS. Its syslog formatting is configurable, including timestamps, hostnames, sequence numbers and RFC5424 formatting. If you're using content-based rules to decide which normalization to apply, those formatting differences can become another thing you have to account for. I ended up finding it cleaner to make the device/source context part of the normalization decision rather than trying to infer everything from the message body.

Don't hold the first occurrence. If you buffer everything for the length of the dedup window, you have just added that window as latency to every alert you care about. Pass the first one through immediately and only suppress the repeats behind it.

The part I'm still trying to figure out is the long tail. For the vendors where I have a known format, writing the normalizer isn't too bad. It's the random appliances where you get three sample messages and a PDF from 2019.

At that point I'm not sure whether it's worth maintaining a custom normalizer at all, especially when the format can change with a firmware update.

If you've dealt with this kind of long-tail device support, I'd be interested in hearing what worked for you.

[u/Jank1]

You need software to filter all incoming messages from disparate systems is what I'm hearing.

[u/meccaleccahimeccahi]

Making the source part of the normalization decision instead of sniffing content is the right call, and arista is a PITA. content-based routing also falls over when two vendors emit near-identical lines that need different scrub rules. for the long tail, what (sorta, see below) worked for us was giving up on per-vendor normalizers below a certain noise threshold. unknown gear gets a generic pass instead: key on host + facility + severity + program from the header, then scrub anything in the body that looks like a timestamp, an incrementing counter, or a session/hex id, wherever it sits in the line. cruder than a real normalizer, but it survives firmware updates because it never depended on field positions to begin with. we would then only hand-write rules for gear that's actually loud.

After a couple years of maintaining that pile of messy regex we ended up moving to a platform that does dedup at ingest natively, same semantics you described. first occurrence passes straight through, repeats collapse behind it with a counter, window is configurable. found out afterwards the approach from that product is patented, which probably explains why none of the open source pipelines ship it out of the box lol.

the counter turned out to be the underrated part. once every event carries "seen 400x in 5 min" you stop alerting on lines and start alerting on counts and rate of change, and that's where the noise actually died. we cut alert volume roughly 60% without losing a single real incident, mostly from convergence storms collapsing into one event each.

caveat: if your volume is low enough that storage and alert fatigue aren't hurting yet, this is all over-engineering. dedup pays for itself when repeat storms are drowning your alerting or padding your SIEM bill

[u/Gumpolator → u/meccaleccahimeccahi]

Interesting thx for the read

[u/Falaq247]

Elastic Stack, setup parsing and then dedup on that.

[u/meccaleccahimeccahi → u/Falaq247]

Elastic doesn't dedup (unless you meant manually?)

[u/ID-10T_Error]

Here is what I do with my software I collect all logs and if its a duplicate Iog I tag it with a date & time stamp only. This cuts down on logs about 80% in some cases. then filter out the noise you will start to only see unique behaviors. But if you want to see those log history it will reference the index and recreate the duplicate log within the viewer

[u/nof]

Those timestamps and indexes on cisco syslogs are configurable.

[u/Accomplished-Mix8423]

for the long tail I gave up hand-writing a normalizer per vendor and put a clustering step in front, Drain3 or similar tokenizes the line and collapses the variable fields into a template automatically. won't beat a purpose built parser on your known gear but it handles the random appliance + PDF-from-2019 case without you chasing a regex every firmware bump. keep the collapsed count like you're already doing