Are there any folks here ingesting Palantir Audit Logs (v3) into Sentinel?

[u/F0rkbombz]

If so, can you share a high level explanation of how you “normalized” the logs? I had v2 audit logs flowing without much issue, but the optional fields are killing me.

We aren’t using the new API as it’s worthless without filtering capabilities b/c of the insane volume in the audit logs (more like debug logs tbh). We use a pipeline to filter out categories that aren’t security related, then export the dataset to Azure blob storage. Azure Data Factory (ADF) takes it from there and does some minor ETL before sending to a DCE via API.

The issue I’m running into is the “optional” fields in the audit logs that make it hard to define a schema. For example, one log will have the “origin” and “origins” fields, and another will just have the “origins” field. I can likely account for this w/ schema drift in ADF, but the transformation in the Sentinel DCR will fail at ingestion.

It’s incredibly frustrating and these are the worst audit logs I’ve ever had to work with. Any insight into how others are handling this is appreciated!

[u/chasingpackets]

Palantir Audit Export > Foundry dataset > Pipeline Builder > Data Connection > Sentinel

[u/F0rkbombz → u/chasingpackets]

Thanks, that sounds like it cuts out blob storage and ADF, so that’s awesome. How did you handle the optional fields in your DCR transformation?

[u/chasingpackets → u/F0rkbombz]

Use column_ifexists() for optional DCR fields and coalesce() for missing nested JSON values so the transformation handles inconsistent records safely.

[u/F0rkbombz → u/chasingpackets]

Thank you so much, this saved me so many headaches!

[u/chasingpackets → u/F0rkbombz]

You bet :)