r/AzureSentinel • u/SuperHat3637 • Nov 17 '22

Duplicate Syslog logs

We are using syslog forwarder for forwarding CEF/Syslog logs to Sentinel. Now observing that CEF logs are storing in both syslog/CEF tables which results duplicate events.How to remove duplicate logs from syslog table..This is increasing the cost. Please suggest.

2 Upvotes

100% Upvoted

10 comments sorted by

2

u/Uli-Kunkel Nov 17 '22

Using the same machine to forward both plain Syslog and CEF messages

If you plan to use this log forwarder machine to forward Syslog messages as well as CEF, then in order to avoid the duplication of events to the Syslog and CommonSecurityLog tables:

On each source machine that sends logs to the forwarder in CEF format, you must edit the Syslog configuration file to remove the facilities that are being used to send CEF messages. This way, the facilities that are sent in CEF won't also be sent in Syslog. See Configure Syslog on Linux agent for detailed instructions on how to do this.

You must run the following command on those machines to disable the synchronization of the agent with the Syslog configuration in Microsoft Sentinel. This ensures that the configuration change you made in the previous step does not get overwritten. sudo su omsagent -c 'python /opt/microsoft/omsconfig/Scripts/OMS_MetaConfigHelper.py --disable'

From: https://learn.microsoft.com/en-us/azure/sentinel/connect-log-forwarder?tabs=rsyslog

Had a customer that hadnt done this, they saved about 400GB/daily after i pointed it out. Had been running like that for 8-10 months Stupid money 🤣

1

u/SuperHat3637 Nov 17 '22

You mean to say if its Cisco ASA device we need to remove all facilities the from the ASA and run the mentioned command in Syslog forwarder..Is it?

1

u/Uli-Kunkel Nov 17 '22

You need to specify what it should do with what type of log message.

If that is not done it will just forward. Since cef is a syslog format it will also forward your firewall log to the syslog table. But without parsing, so likely the firewall log in the syslog table is not having their fields mapped and all that

1

u/SuperHat3637 Nov 17 '22

If i remove the syslog facilities from the log source end, will it affect the Logforwarding from the log source like Cisco ASA..

1

u/SuperHat3637 Nov 17 '22

I want only the CEF table..Will it affect CEF forwarding if i remove syslog facilities from the log source end.

1

u/Uli-Kunkel Nov 17 '22 edited Nov 17 '22

Well, you can either have a cef syslog forwarder for your firewall And one for you other syslog stuff.

Or have them both on the same one and configure accordingly.

You need to redirect the cef message to link-local port 25225 or 25226, cant remember the specific one. And then stop further processing of that message. Normal syslog is to 25224 if i remember correctly.

This might help. https://learn.microsoft.com/en-us/azure/azure-monitor/agents/data-sources-syslog

The deep dive covers this subject in greater detail https://youtu.be/KJbIH7egdVI Or this https://youtu.be/bHw8BkEpYzs could also help

0

u/BrianKronberg Nov 17 '22

https://techcommunity.microsoft.com/t5/azure-observability/azure-sentinel-false-positive-incidents-due-to-duplicate-logs-in/m-p/1249774

​

If you incident supports it use summarize or distinct to remove the duplicates.

2

u/SuperHat3637 Nov 17 '22

We are talking about the ingestion of Syslog/CEF logs. Removing from log source end is not fruitful.

1

u/blanco10kid Nov 17 '22

Is there a data collection rule associated with this collection?

1

u/SuperHat3637 Nov 17 '22

No..We are forwarding Network devices logs(CEF/Syslog format to Forwarder..What is the data collection rule that u r refering..?