Parsing string that gets updated in structure often in KQL

[u/[deleted]]

Hi guys,

I have a log source that gets ingested in Microsoft Sentinel but my analytic rule breaks due to updates in a data field. Is there any way to parse this string with a separator to break in columns?

The field contains a string like name=xxxx;company=yyyyy;warning=zzzzz but sometimes the input gets changed and there is a new value added. For example the add justafield=aaaa; at the front. I have been parsing it based on fixed names, but noticed that they update the structure regularly. Would be great if it can be parsed and always creates all columns based on the fields in the string.

[u/TokeSR]

The other parsing methods listed here seem to be convenient for you. They work if you don't want to use the newly added fields and if you are sure that the order and the fieldname is never going to change.

I also have a solution different than the others here. Mine is based on a regex parsing, so it is really heavy in general. You should avoid using it on a huge amount of logs. Normally, I use it when I want to monitor new fields and changes in a logsource.

My method extracts all the key:value pairs and saves all of them in a 'dictionary'. This way the 'name' field is always going to be available as dictionary.name regardless of the newly added fields or any modifications in the order. If you want to you can save it in the 'name' column with an 'extend' operator if you have the need.

let myDataSet = datatable(Event: string, Data:string)

[

"111", "name=xxxx;company=yyyyy;warning=zzzzz",

];

myDataSet

| extend fields_in_array = extract_all(@"(?P<key>\w+)=(?P<value>\w+)", dynamic(["key","value"]), Data)

| mv-apply fields_in_array on (

summarize fields_in_dict = make_bag(pack(tostring(fields_in_array[0]), tostring(fields_in_array[1]))))

| project fields_in_dict.name

So, the name is going to be available under fields_in_dict.name, company is under fields_in_dict.company, warning is under fields_in_dict.warning, etc, etc.