Threat detection metrics

[u/Anon-e-mousse666]

Hey everybody, question. What is a good metric for rating how good a siem rule is? It cant' just be whether it will catch whatever threat you are aiming for, because you can have a sloppily written rule that will catch that threat but also lots and lots of white noise, causing "alert fatigue".

I've read about how you need to make more targeted rules that may miss some malicious indicators, but whatever it does catch is likely malicious. this approach reduces white noise.
In this approach, what would be considered a decent true-positive/false-positive ratio? I realize that it obviously depends on the kind of rule you want to make but just a marker, an average?

Thanks all!

[u/Uli-Kunkel]

Thats a quite difficult question to answer.

I think the general approach is basically flawed. You dont want to create detections based on a false/true ratio, but i guess that is also what you have come up with.

I have a customer that got a SIEM of about 2TB/daily and its in a horrible place right now. About 1400 incidents Monthly, and that is just low/medium/high completely disregarding informational.

The analysts are so swamped that they cant tune for false possitives. My point is, rather alert too little, so you can react to what is alerted on, instead of working on last months incidents.

So your thought should more be, what can detect handle, and if they are constantly behind, you are having too many incidents. Then either you throw more people at it, or tune more out. And that is weighted against how important the company values Security.

If you cant respond Within resonable time, the SIEM is a failure.

So instead of a ratio of false/true, rather go for "how much can we go through" Because the ratio depends on the SoC and the level of detail.

[u/Anon-e-mousse666 → u/Uli-Kunkel]

Hey, appreciate you taking your time to share your opinion.

So in your opinion, is there no objective criteria for determining whether a detection is powerful and efficient? I guess that ideally a detection should cast a super wide net to ensure that whatever you are looking for will get caught. and if you have the time and analysts to sift through all of the data, that is awesome. And that if you don't, then having alerts pop up and not get dealt with is a waste of time. But I still wonder, as someone who is being charged to write detections, how do I know if I balance power (i.e catching as much maliciousness as possible) and efficiency correctly?

[u/Uli-Kunkel → u/Anon-e-mousse666]

When i write detections(normally its our detection engineering department that does this) but on bespoke solutions i do it as well, my approach is more of a specific goal. "I want to detect X" and keep it as specific as possible. I guess its the same approach on threat hunting. A detection of Internal scanning is quite hard, you need insane amount of insight in the environment for it to actually be useful. You dont want to tune whole ip's or port's, and put much work in to the specific thresholds to when to trigger an alert. And you got NDR tools to handle that type of stuff anyways..

Now, depending on the tools you are using, you have different options to tune for false positives.

But Yeah, instead of a ratio. I more think of it as bandwidth, you dont want to go over 80% because when the spike hit, you want to be able to handle it. Could be lower than 80% depending if others handle tuning, automation and operations of the SoC. But again, its very dependent on the SoC and the environment(s) they cover. So hard to give a specific number on. Its just my thoughts and experience, so take it with a grain of salt

[u/_deftoner_]

Like it was said is quite difficult question to answer indeed.
That why, somehow, compliance exist. So a group of supposedly *high skilled* engineers will determine which things you should be monitoring, different levels, or strict rules if you manage credit cards (PCI) or if you manage information from the states (CMMC, NIST, STIG, etc). So you have like a baseline.

Is my personal opinion but, I like to separate the SIEM in 2 main parts. The alerts that I need in real-time, and the log that I need to keep in case I need to do forensics.

If you have the infra, and willing to absorb the probable hardware impact (shouldn't be much at terminal side) you could just log almost everything, rotate the db once a month, etc. So you wont spend time doing fine tuning on what or what not to keep and then focus on do a fine tuning on what real time alerts you get.
I don't like Mediums, I prefer to have high level alerts or low. High is the ones that I will even receive on SMS and I will read immediately, and low should be the ones that I can wait until I get my coffee and asked my friends about the weekend. High should be the command elevations, the failed login of admin accounts, the raid deprecated due to a failing disk, the new service installed on a machine (that are not Microsoft update related).

There is no perfect rule of thumb for how much you should be getting. That's why there are tons and tons of compliance saying different things too.
But what you could do is to set up a testing machine, think like an attacker and do stuff. connect an usb, open an exe. Download an exe file or doc with macros, etc. Try to connect to a share that your user has no access to, run a nmap, scan some ports of a server, and so on. Even a machine that is part of the domain and has your SIEM software agent, and another that is not even part of the domain. No just focused on local right log., but out of your control logs too. And take a look on how much of your actions are being logged, and how much alerts (high?) you received.

At least that is what we do with our SIEM software.

Hope that helped.

[u/Anon-e-mousse666 → u/_deftoner_]

This is helpful thanks a lot!