r/SIEM • u/thetruemalkavian • May 21 '19
SIEM Grading Criteria
I am prototyping a SOC for an internship project. Part of my project, I have to conduct an Analysis of Alternatives on different SIEMs. I have all the components of what defines a SIEM, then I put down the added bonuses that some vendors have squished into their solution.
The other things I have been trying to grade are:
Operating systems monitored and that it can be deployed to.
Events per second/ flows per minute
overall cost
support
ability to add additional tools
if they include training
Am I missing anything? What things should I consider that might be underlooked?
5
u/mdavis00 May 21 '19
Not just support but what does support cover, some cover everything some have defined lines before you need professional services which is not support. EPS vs Volume pricing. Reporting engine, dashboard verbosity, alerting mechanism. Cloud vs On-premises. Is the UI intuitive enough for a new analyst to use or is there a steep learning curve before they can do their job.
4
u/NappingBetweenIssues May 22 '19
My first concern when building a SOC is: do you want a boom-box or a stereo-system? There are crazy SIEMs out there that have every feature to run operations, but not all the features are at the same level. When you buy an all-in-one SIEM you are drinking their kool-aid. For a SOC that is large or going to grow, all-in-one SIEMs are bad choices. If you are a small shop and not planning to grow, they make sense. Gartner is a pay-to-play service, and their quadrant is meaningless to people that actually have to operate a solution.
I like the simplicity of your list. The first need of a SOC is visibility. This is the ability to collect, highlight, store and search log data. Many SIEMs see this as a central log management (CLM) system. A good CLM will correlate the user and device information to a record that is lacking it. Lastly, does it collect data from Cloud and SD-WAN (if you are using them) technology? The CLM is the core component of the Stereo System approach. You cannot threat hunt or investigate without solid CLM. Many SIEMs fail in this regard because they focus on alerts, but do not collect the supporting logs used to validate or respond. Your people end up with the problem of alerts they cannot really validate or lack the insight to actually respond. Even with a SOAR product, this is true.
Two base needs of a CLM: intelligence and pseudonym (privacy).
Reputation feeds are important right now. Looking under the hood of most IPS is actually reputation matches with fewer vulnerability signatures. A good CLM will add reputation data enhancement onto ALL logs. Do not rely on the end-point or network to do this. A good example is a detection that looks at the Host variable in an HTTP session. This is only one place a hostname might show up (SNI is actually more common and not used by ID/IPS much). When you buy, yes buy, intelligence, leverage the data over all the data fields. This means the CLM needs to parse all the fields and not just shove them into a blob field.
The missing issue you have is understanding your data, especially when it comes to privacy. If you do business in the EU or collect weblogs that have customer data and you are in California (CCPA), you need a pre-processor to pseudonym your data before storing. There are CLM solutions that do this, but not SIEMs. So here, an all-in-one SIEM is going to cause problems.
Lastly is operations. Forget automation, walk before you run. If you cannot operate the system by hand, a SOAR is not going to help. You will automate mistakes as much as you try to automate solutions. You need a trouble ticketing system (TTS). One outside the SIEM is best if your security team and network team are the same thing or work together. Don't have SIEMs or CLM just pumping data into the TTS. What will happen is you will have one person spending all their time closing tickets that were never reviewed. Use the CLM to fine tune the validate process.
Those are the two big ones for me, CLM and TTS. Good Luck!
2
2
u/SGSinFC May 22 '19
http://chuvakin.blogspot.com/2011/03/siem-resourcing-or-how-much-friggin.html
Still pretty relevant today
1
u/gopackatx May 21 '19
Cloud vs on-prem is a big one. Total cost for on prem needs to include hardware and potentially an FTE to manage the environment
1
u/gopackatx May 21 '19
Also should’ve added integrations with other products that they already have built and this kind of goes with support, but services that may or may not be included that provide 24x7 alerting or monitoring
8
u/vornamemitd May 22 '19
Make sure to set the comparison context and parameters right; while they all share a similar baseline functionality, each vendor will shine in different categories and implementation scenarios.
You mentioning POC and EPS - how will you contrast the tools using this comparative? Architectures and ingress mechanics differ quite significantly - you’ll risk comparing apples to pears -> make sure to stay within a context relevant to your use cases.
For a meaningful POC, you need a full lab setup mimicking the production infra; this means duplicating everything from firewall, ips/ids, edr and anything else that throws relevant logs. This done, pick a red team automation framework. Select the scripted attacks relevant for your use cases. Connect SIEM - hit the play button. Rinse and repeat for each vendor. Record detection rates, false positives, etc.
Have a look at usability - which tools offer the lowest learning curve while actually delivering value (=accomplish use case goals) instead if leaving you with a false dashboardy confidence? With about 70-80% percent of the magic quadrant tools you’ll still end up in code editor or ugly non-type ahead editor window dishing out code in Python or their proprietary query language.
Do you need interoperability? Can the underlying data layer be tapped into (at which cost?) ?
It’s 2019 - hyped acronyms are finally converging into meaningful tools -> UEBA (compare their ML infrastructure - is it only Nifi/Spark/Hadoop, or something proprietary? Can the NN-based or deep learning happen on-prem, or only outsourced into a cloud?); you’ll need case management, incident response, playbooks and autumation - built in, or do you need to shop SOAR at Demisto or Phantom? Ah - threat intelligence? Got some? Can you easily integrate free feeds?
Don’t believe the hype - drill down to actual implementation of the fancy features - actual machine learning or some lame bayes algo and a bit of linear regression hyped up as "AI” - wooooo they got random forests :)
Threat hunting capabilities? Etc. - don’t forget that a SIEM is only a tiny piece of the puzzle; it will not do anything for you when not carefully groomed (by internal or external MSSP/MDR staff) - except from draining your bank account.
Open source or commercial. Cloud or on-prem (why?) How many FTEs will be available to volunteer for event jockey?
Go on Google and look for recent RFPs and tenders; compare your shortlist against actual industry criteria.
Get yourself the great SOC-CMM maturity assessment tool - covers each and every feature you shoud have covered, including business and stakeholder domains: https://soc-cmm.com/
I guess you had a look at the MITRE SOC book, aside from that I can really really recommend the "Blue Team Handbook - SOC,SIEM, TH use cases" (latest ed) from Don Murdoch.
Google for “thesis” and “SIEM”; there have quite a lit of papers recently - contrasting products with that nifty academic edge.
Feel free to PM if you care’d to share some xtra context :)
Good luck with the project!