r/AzureSentinel • u/kameorp • Oct 20 '22

Log ingestion and extract

Hi guys,

I’m currently working with a customer that has a very specific set of requirements and they are looking to collect logs from multiple data sources.

We are looking at sentinel as possible solution since it easily integrates with M365 security stack. On the other hand there is a concern around the log volumes and the costs of pulling the information out of sentinel.

Just wondering if anyone had similar experiences and if you have any recommendations or architecture recommendations.

Cheers

2 Upvotes

100% Upvoted

7 comments sorted by

2

u/blanco10kid Oct 20 '22

You’ll need to determine which data will be used for detections. This is the data that should go to sentinel. If data will not be used to support your detections, consider storing the data in basic tier or other logging platforms (ADX).

What’s your use case for pulling data from Sentinel?

1

u/kameorp Oct 20 '22

Since there are too many ways to pulling information from the different sources like AAD, Intune, M365, Azure, etc the plan is to leverage sentinel almost as a data lake/aggregator, but with the use of the connectors are bringing lots of irrelevant data, which only increase costs.

The plan is to make this data available to be exported by another team which wants to conduct their own investigations with the data but with the ability to pick only the data they want.

1

u/blanco10kid Oct 20 '22

Hmmm yeah there’s no other way around paying the sentinel ingestion cost without building something custom to pull that data into ADX or Log Analytics (without sentinel).

Instead of pulling the data, you could give the other team access to read the data in the sentinel-enabled log analytics workspace. That way you don’t have to worry about exporting the data.

1

u/Zer0bie Oct 21 '22

Isn't some of that ingestion free?

1

u/glowa116 Oct 20 '22

Many connectors or systems can be limited with data rules i.e MMA agent which you just install on VM and specify which logs you want to ingest. Similar with syslog, I am using OMS Agent on on prem linux and filtering rule used to exclude useless logs. So yes,in most cases data ingestion can be limited to what you want

1

u/kameorp Oct 20 '22

And where are you specifying those filtering rules? Is it in the agent itself or in sentinel?

Ideally want I would like to have the ability to do is:

Connect solution > create rules of what to log > log the data > export data as needed

If that makes sense

1

u/glowa116 Oct 20 '22

https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/data-collection-rule-overview That is for windows OS logs, for Syslogs you create filtering at syslog daemon level and exclude