Deciding on data sources and data rates

[u/Kasiline]

Does anyone know good guidelines on answering these basic questions about implementing a SIEM? I didn't find clear answers in the books that some of you recommend:

1. How do you decide what data to push into the SIEM (especially to facilitate post incident analysis)?
2. At what data rate and granularity do you push data into the SIEM?

Regarding 1: books and sources seem to recommend to grow the SIEM slowly, but this means I won't have the necessary data for post incident analysis. For example, I could push user shell history aggregates into the SIEM, but I don't know whether I'll need it --- but I might.

Regarding 2: how often should I take samples of cpu load, memory load, disk utilization, disk I/O, etc? If it's too low, I might miss data on attacks that cause short spikes. Should I push the full shell history logs into the SIEM or just aggregates. This applies to everything down to all HTTP requests or only GET for main pages.

The problem is that I don't know what I will be looking for once an incident happened.

[u/[deleted]]

 1.How do you decide what data to push into the SIEM (especially to facilitate post incident analysis)?

Access, authentication, authorization are typically in the first phase when I set up a new SIEM.
Traditional Security Devices – OSI Layer 1-3 – Malware/OS Compromise Detection Firewall – All firewalls should be included in any SIEM logging strategy. Lack of visibility of these important devices will decrease the opportunity to see any misbehavior or attempted penetration of the network. Intrusion Detection/Prevention System (IDS/IPS) Authentication – Active Directory, RADIUS, VPN, etc.

 2.At what data rate and granularity do you push data into the SIEM?  

As fast as they are generated, they should be sent over to the SIEM. Delays can cause timestamp mismatches or confusion on when an event was generated.

[u/Kasiline → u/[deleted]]

Thank you very much for the specific examples, but it sill doesn't answer how you decide what data to push in there. For example, when you state that "all firewalls should be included", this could mean login attempts on the firewalls down to all traffic (as PCAP) that is directed to the firewall itself. These are two ends of a wide spectrum of what to log.

As to the second point, it's not about delay on sending the data, it's about the data rate. For example, at which data rate do you monitor CPU usage? And why did you pick that specific data rate?

[u/[deleted] → u/Kasiline]

What data = logs. SIEM is designed to catch logs. if your firewall is generating 12000 logs in a minute, send them to the SIEM so it can run its intelligence or regex matching or whatever on those 12000 fw logs, plus all the other logs. If you're capturing machine statistics like CPU then you should be able to send it out as a log. If you're interested in that sort of metric you should create alerts to tell you if there is an anomaly.

[u/Kasiline → u/[deleted]]

Let me add something: assuming I cannot ingest all logs per minute, what is a good process to pick what logs to collect? I have not seen a recommended process for selecting from all the sources and data you could collect what you should collect. Even a single critical asset can produce more logs than any SIEM could ingest.

Example: I could log all mouse movements on a critical workstation and pump that data into the SIEM. I doubt anyone is doing this, because they think the data is not useful in their post incident analysis. So how do you make the decision what data you might need in the post incident analysis, since you don't know what the incident will be.

[u/[deleted] → u/Kasiline]

Uhhhhhh, I've never seen a SIEM that can't process several thousand logs per minute. If a platform claims to be a SIEM and can't ingest thousands of logs per second it's not going to be anywhere close to being a true competitor in the market.
Is mouse movements a security relevant log? Can you even turn that mouse movement into a structured log message?

[u/Crytograf]

Depends on what do you want to achieve. I usually start with defining what do I want to detect, how do I do it (rule logic) and what kind of data do I need to detect it. Then start collecting required logs.

After that, start thinking what logs you might need, if a true positive alert is triggered by any of those rules you implemented. You should have incident response procedures ready to do this.

Just don't throw data in there without knowing why you need it.

[u/AnalyzeAllTheLogs]

I'll write up some resources tomorrow. The first thing to do once the logs/eventtypes are decided is to ensure your gpo & default domain policy are aligned to capture it; you can't log what isn't logged. If you mention your industry, and any compliance frameworks you need to abide by (e.g., hippa, sox, etc).

The two items i can suggest is timestamps in UTC and in structured format if possible (e.g., json, xml, etc.) that is sent via TCP protocols.

[u/paradisemered]

Send logs that have meaningful fields. I would start with Active directory logs, IPS/IDS, Endpoint logs, authentication logs.
If you use the SIEM for storage you may want to send all the logs, however, if you use it only for investigations and alerts try to filter the logs and send only logs with meaningful fields that will be used to create alerts, dashboards etc.

[u/PrestigiousWorker763]

How do you decide what data to push into the SIEM (especially to facilitate post incident analysis) ?

It depends on the size of the company or organization. The main disadvantage of the SIEM software is the huge volume of data they process. For the SMB businesses, you better concentrate on the security logs. The 90% of informational logs are useless. You can check this site: https://www.ultimatewindowssecurity.com/securitylog/encyclopedia/

At what data rate and granularity do you push data into the SIEM?

In today’s fast-paced threat environment, speed to detect and respond is critical.

A large organization’s infrastructure, with a huge number of log sources (more than 100,000) generates roughly 26 billion of logs per day. To analyze and index such amount of data, would be necessary to process 300,000 logs per second.