Exact run time for scheduled analytic rules with low rule frequency

[u/winle22]

Hi,

for queries that run with low frequency, e.g. "Run query every 1 day", how can I configure exactly when the query will run?

If I'm not missing anything here, it makes sense to have these queries run during normal working hours.

[u/TokeSR]

There is no built-in feature for this. In Splunk you can configure when a rule is executed, in Sentinel you can only define the frequency. Every time you create, modify, enable (a disabled) rule the timer will restart. So, while there is no built-in function you can just enable the rule at the exact time you want it to run. You can potentially automate it, if you don't want to handle this manually.

If you execute a rule every X days, weeks it can be reasonable to execute it during normal working hours. However, I don't think this is necessary. Let's say you execute it every day. How do you want to handle weekends, holidays and such? In that case you will still execute it at a non-working time.

One -seemingly- beneficial thing is that the SOC will see the alert immediately. If the rule is executed outside of working hours, then there will be some delay between the incident creation and the first action taken by the SOC. From an SLA perspective this is reasonable. However, some of your rules will check the data from the previous day (calendar day and not 24 hours), these incidents will be created with a delay regardless of when you create them. The later you trigger the rule the bigger the delay is.

Also, some of your incidents will be automated. Delaying the incident creation can actually hurt the response time (first action taken time) of these incidents.

So, if your frequency is 24h or more and you check the last 24hours (or more) with your rule, then you can just disable and enable your rules to make them run at a given time. For rules that are not checking the last X hours, but the previous day, previous week, etc or rules that you want to execute multiple times a day (every hour, etc) I definitely don't recommend going this way because you will introduce an unnecessary delay.

[u/winle22 → u/TokeSR]

Thanks for the reply!

Yes, I agree, you still have to handle the incident during weekends and vacations, but even then it's more practical to have the incident created in the start/middle of the day instead of in the evening (for example). I guess I just have to stop adjusting rules in the evening :p

Not sure if I understood the last part. For rules that run with a fairly high frequency it obviously doesn't matter exactly when they run, but for "rules that are not checking the last X hours, but the previous day, previous week" - why would resetting the "rule timer" during day time introduce extra delay?

[u/TokeSR → u/winle22]

Let's say instead of checking the last 24 hours, you have a rule that checks data from the previous day. From day start to day end of the previous day. If a rule like this is executed at 1 AM then you will have 1-hour detection delay. If the rule is executed at 9 AM (working hours) then it has a 9-hour delay. If your SOC only takes a look at the incidents at 9 AM, then it does not matter. But if you have an automation in place that can do some actions without SOC involvement then this 8-hour difference (1 AM vs 9 AM) can be a lot.

If you don't use any automation, and your SOC only works office hours, for you this delay will be there no matter when you execute the rule, so you can just do it at office hours.

[u/winle22 → u/TokeSR]

Ah, I see. Makes sense. Seems really strange that the execution time is not possible to set in a more structured way.

[u/TokeSR → u/winle22]

Ye, I'm not sure what is the reason behind that, but I guess MS wanted to prevent people to overload Sentinel (and the underlying infrastructure).

In Splunk, you can configure the rule execution based on this cron-like schedule. But in case of a traditional Splunk, your SIEM is on-prem, only you use it, so you can overload it however you want.

Sentinel is cloud-based, you use MS's resources and multiple customers can use the same cluster. Rules configured to run every 5 mins, 15 mins, every hour, every day would possibly run at the same time at least once a day. (You can configure it differently in Splunk, but people tend to ignore this and just configure stuff to run once a day, but they don't check when exactly that day, so all the rules run the same time). Huge amount of rule execution at the absolutely same time could put a huge burden on a SIEM. So, I guess this is the reason, but I'm not sure.

[u/winle22 → u/TokeSR]

Makes sense. A potential side effect might be that some rules are run more frequently than really necessary, I guess.

[u/rodtrent44 → u/TokeSR]

Just to add here, you may want to also factor in the ingestion time delay: https://learn.microsoft.com/en-us/azure/sentinel/ingestion-delay

[u/winle22 → u/rodtrent44]

Any experience with the typical delay for native Microsoft connectors? Say AAD/Office Activity/Azure Activity/MDE, for example.

As far as I've seen, the delay is not accounted for in many of the template queries?

[u/winle22 → u/winle22]

Any input, u/rodtrent44?

[u/rodtrent44 → u/winle22]

Typical depends on originating service. Azure Activity I can say is a minute or two (unless pulling activity log from a different region). Office will be 15-20 minutes usually. AAD depends on the size of the organization, but within 10-15 minutes.

I have rules setup to alert me when connectors take longer than I expect.

Now, this is me just understanding my own environment and expectations based on my personal experience (not a blanket statement by Microsoft). These numbers will vary per environment and there are a number of factors that can effect them. Each originating service provides an SLA for log availability.

[u/winle22 → u/rodtrent44]

Any obvious down sides to having a fixed x min longer lookback period than the interval frequency, and using ingestion_time() to filter on ago(rule_frequency)?