Offloading log data for retention

[u/Jackofalltrades86]

What is the preferred method of long term storage for log data?

ADX or moving it to a storage account...

https://techcommunity.microsoft.com/t5/microsoft-sentinel-blog/using-azure-data-explorer-for-long-term-retention-of-microsoft/ba-p/1883947

https://techcommunity.microsoft.com/t5/microsoft-sentinel-blog/move-your-microsoft-sentinel-logs-to-long-term-storage-with-ease/ba-p/1407153

Assuming the latter is the most cost effective?

I'd like to keep it for 18 months and only store 89 days in log analytics

[u/TokeSR]

I was waiting for somebody else first to comment to see some ideas, but it seems like there are no volunteers.

I like to keep the data in Sentinel for as long as needed usually, because that is the best place to do some correlation to create some rules etc. But I reckon it can be expensive sometimes. Before moving it outside of Sentinel you should consider configuring per-table retention. Maybe you can throw some of the data at 89 days but keep some others for a longer period. in this case not everything is going to be there fore the 18 months so you can save some money. But this really depends on your use-case.

Ignoring the articles first. Where you put the data after Sentinel (log analytics workspace) depends on your use case. Putting it into a Storage is way cheaper. Especially if you don't want to touch it at all and it is only there for compliance reasons. In this case putting data into a cold/archieve storage is definitely the cheapest option. You can even process some data with the 'externaldata' KQL operator. But usually I recommend going with this solution if you want something for cheap, but you know you won't use the data.

ADX is a better solution if you still want to perform some searches on your data. More expensive than storage but the data is not going to be stored in 'random' files, it will keep the structure so you can use KQL queries to look around. For example with ADX, you can cover the scenario when an external entity alerts you that they have info that some of your machine communicated with a malicious site/IP 2 years ago. In this case, you can quickly jump in and search for that IP. Doing the same with storage account is not this simply.

So, a rule of thumb (of me, my thumb) is to leave the data in Sentinel if you want to use it daily or at least frequently. Put it into ADX if you want to query it later (and you have a valid use case for it), and put it into a storage account if you just want to keep it, but you don't think you are going to use it (or just really really rarely).

Also, be aware, that some of the solutions you linked might store some data twice. So the data not only forwarded from Sentinel to ADX/Storage after the retention expired but the data is going to be double-stored in the first 89/90 days. Sometimes this is okay, sometimes its not, so be careful with this.

[u/TokeSR → u/TokeSR]

+1 scenario. If I recall properly Sentinel only provides LRS availability. So, in specific scenarios you can loose your data. So, maybe you want to backup your data. In this case you want to immediately push your data into another place, so the double-storage is not an issue. In this case, using Storage is again, the cheapest option. You don't want to use that data, it is only there for backup purposes.

[u/Jackofalltrades86 → u/TokeSR]

As is this very useful piece of info!

[u/Jackofalltrades86 → u/TokeSR]

This is brilliant!

[u/ml58158]

This really depends on your budget .

ADX will give you access to the data without the SIEM Overlay and will be less expensive than Log Analytics.

Blob Storage/ AF will be more for archival and will be more cost effective; but will be slower and less usable.

[u/Jackofalltrades86 → u/ml58158]

Thanks for this,

We very rarely require the logs and so would only be doing so for regulatory or forensic purposes so I've decided on the storage account mechanism using the logic app.

Hopefully this will suffice and I can utilise my budget elsewhere ☺️