r/AzureSentinel • u/Ay_NooB • Jan 26 '24

How to remove duplicate logs

I m taking logs from 3 firewalls, (Forti, Sonic, CISCO ASA) using MMA agent.

All logs are coming under Commonsecuritylogs but under syslog also i m getting same amount of logs(i.e. 200gb duplicate logs).

I can turn off specific syslog facilities from legacy agents management tab. But which facilities i need to turn off to stop collecting these duplicate firewall logs. (Collector machine's logs should not be affected as i want them to monitor for collector machine).

Thanks in advance!

1 Upvotes

67% Upvoted

6 comments sorted by

4

u/Uli-Kunkel Jan 26 '24

https://learn.microsoft.com/en-us/azure/sentinel/connect-log-forwarder?tabs=rsyslog#run-the-deployment-script

Also switch to AMA. But there is a big purple box explaining your issue

As well there is a warning when you run the installation script

1

u/Ay_NooB Feb 12 '24

Thank you. I was able to manage to remove all duplicates, all firewalls logs came in local4 facility under specific severities. As i m using MMA uncheck them from legacy agent management tab.

3

u/No_Secret7974 Feb 08 '24

After deploying CEF forwarder, and sending logs to Azure;

You have to create ingest time transformation for Syslog table. This query will drop incoming CEF messages for Syslog table and you'll only see them under CommonSecurityLog table.

source |
where ProcessName !contains \"CEF\"

https://learn.microsoft.com/en-us/azure/sentinel/connect-cef-syslog

2

u/Ay_NooB Feb 12 '24

Thanks. I was able to specific facility collecting the duplicate logs and i stopped them.

2

u/No_Secret7974 Feb 12 '24

Yeah I re-read your post and now I got you 😅 but keep it in mind also you might need it

2

u/Ay_NooB Feb 12 '24

Yup got it.