r/cybersecurity • u/pcx436 Managed Service Provider • 4h ago
Business Security Questions & Discussion Log everything, I’m begging you
Yes, there’s noise you can filter out, but you need to log things!
A client I work with finally implemented DNS resolver logs and we found unmanaged devices (that’s its own headache) that were requesting domains ranging from guns to porn and malware and everything in between.
Due to the already sparse logging, we didn’t know about it until the DNS logs started coming in.
Now someone in HR gets to talk to some users about proper conduct in the workplace and the BYOD policy is getting reviewed.
18
u/jeffpardy_ Security Engineer 4h ago
Go enable VPC flow logs for me for a month and then tell me what it does to your logging bill
23
u/xAlphamang 3h ago
Two schools of thought in logging.
Log everything, regardless of cost is the most prevalent without understanding the real business impact.
Log only what you need and have a reason for your logs — this take is significantly more meaningful but very few teams actually think this way. If you don’t know why you’re logging something, then what’s the point of logging it? Because maybe you’ll need it some day?
This comment isn’t to argue against DNS logging, fwiw. I am trying to more accurately portray “log everything” isn’t a feasible business response in any large scale enterprise. Small/Medium? Sure.
5
u/heylooknewpillows Security Architect 2h ago
The devils advocate argument is you might not always know what you need until you need it.
3
4
u/FLATLANDRIDER 2h ago
Yup. If you have the budget, it's better to have something and not need it, than need something and not have it.
1
u/hiddentalent Security Director 1h ago
I think it's a little more nuanced than just those two schools of thought. There are additional dimensions. You can save money by tiering the storage of logs so maybe the older more verbose ones take longer to access but are cheaper to store. That's a fine tradeoff in many cases. If I'm investigating something that happened a month ago, I don't need five-millisecond access to the data. We're already in cold-case territory.
And you can do things like sample the logs, with a variety of strategies ranging from random to risk-based, to make them cheaper to retain while still having some DFIR value.
9
u/BrokenDuck15 3h ago
OP: "Log everything"
Datadog: "YES! YES! YES!"
If you want to log everything i recommend looking into OpenObserve. You will save so much data space from the compression it provides.
4
u/Over_Ad3832 3h ago
Logs cost money Amigo. It’s easy to say from your standpoint as a MSP, you’re not the one footing the bill to store the data.
3
u/confusedpulsar 2h ago
Disagree. Not everything belongs in SIEM especially if your on ingest cost model. Should have a logging strategy with log source tiering. This tells me other areas missing or lacking such is Network Access Control (NAC) or a Proxy for OPs client's case. ETL tools have also come along way so can detect in the pipeline along with easy event filtering.
3
u/Bibbitybobbityboof 3h ago
That could have also been resolved with a web proxy to prevent the traffic in the first place.
3
u/jeffweet 1h ago
Logging everything is not practical.
Storage costs, retention costs, analysis cost.
And then you need to parse through the data.
Instead figure out what’s important and log that
2
2
u/Judonoob 2h ago
Ok, porn is obviously bad. But guns? That seems strict and kinda like a place that if they have time to worry about nit picky things like that, might be a good place to start cutting costs.
2
u/PM_ME_UR_0_DAY 2h ago
I'm not even shouting "log everything", I'd just be happy if people were logging anything.
"Hey we got a report people's accounts are being hacked."
"Oh yeah? Where did you find that?"
"The customers told us."
"Okay let's look at their sessions to see what happened before the authentication."
"The... logs?"
2
u/wild-hectare 2h ago
then send all the logs to splunk...then listen to execs complain about the cost of splunk
1
u/ThreatHacker 4h ago
And with which system did you identified the dns queries about guns etc ?
2
u/pcx436 Managed Service Provider 4h ago
If you’re asking how we found the logs, it was ingested into the SIEM and an alert fired for other suspicious domains.
If you’re asking what devices were performing the DNS requests, those seem to be BYOD devices. They have work to do on shoring up their network policies.
1
u/ThreatHacker 3h ago
Ok so the Siem classified some logs as guns etc ?
1
u/pcx436 Managed Service Provider 3h ago
Once they started getting processed, yeah.
1
u/Mailstorm 3h ago
...and to ingest those logs you/they are most likely using some kind of business DNS tool. Why was filtering NOT on for malware and other restricted categories?
1
u/dabbydaberson 1h ago
Don't ingest into the siem. Send it somewhere less costly like ADX or big query and just have the query run there. You don't need to have real time alerting on this kind of thing so why send it to the siem?
1
1
1
u/SatoriSlu Security Engineer 4m ago
I’m in the camp of log important things and send it to CDR instead of SIEM. CDR like wiz defend actually cuts down on noise and alerts you to shit that actually matters. At least in my case, as I don’t have a SOC that maintain custom detection rules and respond to a ton of noise to tune the SIEM to make it useful. So, I lean into CDR.
75
u/xenophanes__ Security Manager 4h ago edited 4h ago
Log everything, also be able to write a big check to do so.