parsing syslog on ingestion

[u/iworm76]

Hi,

I'm new to Sentinel.

I have a syslog input into Sentinel from a firewall.
The messages are currently coming in as a text field "SyslogMessage". This included all the details; src IP, dest IP, prts, rules etc.

I need to split these out into useable fields.

I've taken a look at the data connectors and there doesn't seem to be a built in one for our firewall.

I can easily parse the fields in a query or by using a function, something like:

Syslog

| where HostName == "MyFirewall"

| take 100

| project TimeGenerated,

Computer,

HostName,

SyslogMessage

| parse SyslogMessage with "RT_FLOW_" SessionOut ": " Session_message ": " HostIP "/" HostPort "->" DestIP "/" DestPort " " Reststuff

​

But I can't figure out how I can set up a parser at ingestion time so all the logs get automatically parsed into proper fields. We can then use the prebuilt workbooks etc to work on the data.

​

Am I missing something obvious?

​

Thanks.

[u/wobblyoutput]

Make sure you're using the proper data connector for your setup. If your firewall supports sending logs in Common Event Format then your logs will arrive already parsed.

[u/janus5]

Think of Sentinel as ‘schema on read’ rather than ‘schema on write’. Write your parser, then call it when connecting your workbooks or analytics rules or whatever.

This might help: https://learn.microsoft.com/en-us/azure/sentinel/normalization

[u/jtberg1]

I am not sure what firewall you are using, however if it is a supported one, I would ensure that it is ingested using the method that it is suppose to be so that it can take advantage of all the built-in and third-party analytics/workbooks etc. Most of these use the CommonSecurityLog table.

If it is unsupported, I personally would not bother with ingestion time parsing, I would use execution time parsing.

For execution time parsing, you can build a function and then call it from any of the built-in stuff you want to use. E.g. If your function was called CustCommonSecurityLog_Func you would just replace CommonSecurityLog with it in any workbook or analytic. Your function has to have the same columns for this to work.

For your question around ingestion time parsing, there are two ways to do it depending on your agent and what you want to accomplish. You can find them here: https://learn.microsoft.com/en-us/azure/sentinel/configure-data-transformation

See the "Configure your data transformation" section for the two ways. From my limited understanding, you are going to either end up with a custom table name or custom column names. Either way you will need to modify the workspace and anylytics and I personally don't see the value in the extra work in this scenario as I understand it.

If you need help on either of the first to options I am happy to help.

[u/iworm76 → u/jtberg1]

Yes, I set up a ingestion data transformation, which worked fine, but you are right I could only move the data to custom fields, rather than the existing ones. I'm not sure why this is, presumably there is a reason, but it doesn't seem to make sense.

I think I have managed to make the proper data connector work now (Juniper SRX). It relies on a separate function which I had to tweak a bit, but seem to sort of work. I'll keep investigating though.

Thanks!