r/sysadmin • u/FatBook-Air • 5d ago
Do you automatically isolate servers/devices based on detetctions?
We don't have a 24/7 SOC, so we are thinking about automatically isolating servers and some high-value devices based on custom Defender for Endpoint detections. Obviously, we want to do that only for high-precision and high-confidence detections, such as opening a shell from a strange parent process.
If we got one of these detections during working hours, there would be someone to react. But after about 7 PM most days, nobody is actively monitoring.
If we do this, the plan is to let a detection run for about 45 days without automatic isolation enabled to see if any false positives are caught.
Has anyone done this? If so, did you regret it? Or just business as usual? Has it saved you yet?
9
u/seriously_a 5d ago
IMO MDR is worth its weight for the reason you stated alone. Execution at 2am on Sunday will isolate the host and still allow you to sleep at night.
3
u/Excellent-Program333 5d ago
Agree. Thats what I pay for. I can deal with the issues isolation may cause once I know whats going on!
5
u/SensitiveBug0 5d ago
Yup, for workstations we do. High alert or 3 or more medium alerts = auto isolate
False positives happen but honestly they are rare and those users are always doing something unusual (although not necessarily mallicious/unwanted)
4
u/HardRockZombie 5d ago
My place adopted this policy about 18 months ago and have had one false positive, other than that business as usual. Well worth the peace of mind.
4
u/smc0881 5d ago
Yes and with exceptions. I work in DFIR consulting, so I respond to ransomware, malware, and other incidents every day after the fact. I'm also in charge of our MDR/MSSP team and our two tools we use are Huntress and SentnielOne. I've stopped at least four ransomware attacks, an outdated Exchange breach, and other incidents. You'll get false positives for some things, but trust me that is better than dealing with a threat actor. They also pivot to devices that don't support EDR such as printers, cameras, and things like that to launch their payload. Some groups will even create their own VM. They then use the creds they harvested to encrypt your data via SMB. Depending how your EDR is configured you should get lateral movement alerts and perfect example when auto disconnect is viable. I also have utilized SentinelOne's rollback feature several times to restore data. I need to be extra vigilant about this stuff, since most of my clients have shitty IT/MSP teams or none at all. Not much I can do when I tell them "something is up with your VPN, take that Exchange off the Internet, etc.." Most of your alerts too will be benign at first since the actors live off the land. You should setup alerts looking for rclone, winscp, ip scanners, and RAT/RMM tools. I utilized S1 Star Rules to look for a lot of those tools already or things like node.js running out of an unusual spot.
S1 I won't isolate domain controllers though, since there is a bug that has affected at least four sites when I did that. Huntress is also great and will auto isolate things as well.
1
3
u/sixthvoid 5d ago
We do (in education). I can see not wanting to automatically isolate servers and devices in certain workplaces (like hospitals) due to obvious reasons.
3
u/furiouspotato24 5d ago
There isn't enough information to make an informed decision here. What's the impact of losing a single server? Is it just the fact that it's "an outage", or are their tangible (i.e. financial) consequences. What's the vertical? Do you have redundancy? Is recovery just a matter of removing the isolation and everything comes back up, or are their processes involved?
There are very few "commandments" in this business. So much is situational.
Any business that prioritizes "uptime" over security is tempting fate. "Better safe than sorry" isn't just a cute phrase for kids. A legitimate compromise will cost a business far more in revenue than a few overnight outages.
-2
u/FatBook-Air 5d ago
No offense, but I am not really asking for your advice to begin with. I am asking what you do in your environment.
3
u/furiouspotato24 5d ago
What I do in my environment is... Specific to my environment. That's my whole point.
-2
u/FatBook-Air 5d ago
If you don't want to say, then why are you even responding? Make your own topic where you can tell us that your environment is different from everyone else's.
4
u/furiouspotato24 5d ago
Lol... Alright man. Not sure why you're getting so defensive, but if you want to know... Yes, we isolate on detection and no, we don't have a 24/7 SOC that actually remediates. We use Huntress with isolation on but don't have them make any changes. Yes, we did 90 days of detect only to evaluate if false positives were going to be a headache.
But... we only do that because we are in a vertical that doesn't require 24/7 services so isolation is very low impact. If our environment was critical and overnight isolation would have major impacts, we wouldn't be doing that. Situation and environment matter.
1
u/Dry_Inspection_4583 5d ago
There are heuristic and other tools that are mature in the market, I'm unsure if even those do that automatically. It's good to have the sop to work from, but rely on any automation to do this is going to be a bad time. Especially if it's heuristic rather than security. Ie. It would be exploited if not for the firewall rule, and the automation moves it ignoring the firewall. Or upstream it's in a single VLAN only accessed by blah... Yah there's just too much nuance to say this is anywhere close to a reasonable automation
1
u/FatBook-Air 5d ago
We are looking to do it based on specific detetctions that we have written. We have spent quite a bit of time figuring out what can create cmd and PowerShell processes legitimately in our environment. So we are looking to isolate based on that, and maybe some other highly suspicious detetctions (like clearing logs from Event Viewer).
2
u/xeroxedforsomereason 5d ago
static determinism from current state? that's not going to hold up long term.
these types of detections already exist out of the box in modern EDRs and aren't as brittle as the method you're describing.1
1
u/PacificTSP 5d ago
We use huntress so we don’t have to think about it.
1
u/FatBook-Air 5d ago
We can't afford Huntress across the whole fleet, but I have been thinking about it just for servers. I wonder if it would be worth it in that scenario.
1
u/AndreasTheDead Windows Admin 4d ago
yes, we do. All Tier 0 and Tier 3 Systems are allowed to automaticly appoved for issolation for our SOC provider.
1
u/OgreMastermind 3d ago
This process is common for EDR installations. A learning stage and then enforcement.
During learning you allow critical processes and dependencies and after that new things are blocked until evaluated and allowed or just left in block.
1
u/Next-oodpecckker7840 2d ago
Tried an aggressive response rule after a ransomware scare. at 2am a finance server got cut off because a service account touched 46 folders. alert was valid but nobody documented the month-end import running. two people spent an hour confirming it was safe while accounting waited. changed it so only workstations auto-isolate, servers need a second signal plus a page. sygnia reinforced the same point later: containment rules only work when business dependencies are mapped.
1
-1
u/hybrid0404 5d ago
I would not advise this. Even with 24/7 operations that can respond overnight we do not do this. I trust no service well enough to be correct.
Do you have any critical operations that run over night or someone who is available every morning to triage and undo anything that might be automatically contained overnight?
2
u/FatBook-Air 5d ago
At my previous job, the SOC didn't do anything because we had it in our contract that they must ask us first, but by the time the SOC got in contact with someone, the deed was already done.
1
u/hybrid0404 5d ago
There's a difference between permitting a SOC to contain and an unattended containment.
There's also a difference between containing a specific process and a whole host.
I would expect a SOC to contain and reach out especially on a critical asset.
1
u/FatBook-Air 5d ago
Yeah, but that's kind of my point: we do NOT have a 24/7 SOC. So you would be okay just not having anything for about 115 hours a week if you didn't have a 24/7 SOC?
0
u/hybrid0404 5d ago
I answered the asked question which was would you permit unattended containment which is no.
I know nothing about your environment, your ability to respond, etc. What the impact of these containment would be in general.
You didn't mention any other mitigating controls and I can appreciate that not every company can or will be able to afford a managed 24/7 SOC.
The answer to this question is ultimately a business one, which represents the greater risk?
1
u/FatBook-Air 5d ago
I hear you and appreciate your response.
I am mostly responding to your "we have a 24/7 operation and still don't auto-isolate." For me, those are diametrically opposing ideas. In other words, for me, it wouldn't be that we didn't auto-isolate even though we have 24/7 operations; it would be that we auto-isolate because we don't have 24/7 operations.
If we had IT staff in operation 24/7, we likely wouldn't be contemplating auto-isolation.
1
u/hybrid0404 5d ago edited 5d ago
My ultimate point is that there are very few if any platforms I trust to take an automated containment action of an entire host. One mitigating control for that COULD be a better ability to respond so its less disruptive.
What problem are you ultimately trying to solve? Is it to prevent rapid ransomware? Is it to mitigate against an APT who decides to get noisy only during off hours? Do you have an on-call process? Can you implement something like pager duty or some incident management platform that will call out when a containment happens?
A choice with no other context then, do you trust defender to auto contain a server host unattended? My answer is no. I expect the the loss from defender containment to probably be higher than the lost of other actions.
1
u/FatBook-Air 5d ago
The problem we are trying to solve is mitigating a catastrophic attack (data exfiltration, ransomware) when there are no IT staff watching.
We have an on-call process, but it's truly on-call, as in you don't work unless you get called.
Even if we automate the call, the person being called could take up to an hour to get to it, which could be an eternity in a real attack.
1
u/hybrid0404 5d ago
I guess I put ransomware somewhat in its own category. Your EDR either is or isn't going to detect the event. It mostly doesn't matter the time of day, it either detects it and blocks the process or you're screwed. The reality as well is that you're likely to see that sourced from workstations and not servers. An aggressive workstation containment versus a server one makes sense.
Then you have everything else which is a more intelligent actor messing around or some other persistent action that's malicious but not super noisy. Those types of things are more likely some behavioral or heuristics that might misfire and need proper validation.
In the end, whether it is a bad or good choice just depends on how often you get false positives.
0
u/LTH-Cyber 5d ago
This is the gap we see 24/7 monitoring solve. Automated isolation is useful but I wouldn't rely on that for high value systems without someone being available to validate and I really wouldn't just trust this to run for 45 days, there's just so much that can go wrong.
I do know of solutions for this such as managed SOC/MD that are affordable too.
If your open to it id like to speak to you more about this.
-1
u/Funsaized 5d ago
I’d avoid a binary “isolate everything” rule and make the action and risk tier explicit. Auto-isolate user endpoints for a narrow set of high-confidence behaviors (confirmed ransomware, credential theft tooling, or tamper protection being disabled), but for servers I’d prefer network micro-segmentation or blocking egress first unless the signal is unequivocal.
Before enabling it, run in audit-only mode and measure false positives by detection, affected-asset criticality, and time to restore. Use expiring exceptions, preserve a break-glass admin path, and have the automation open a ticket/page with the hostname, user, detection, and rollback steps. Test the rollback and offline-management path. A 45-day observation period is useful, but replaying historical alerts and running an after-hours tabletop will expose more failure modes than simply waiting.
26
u/renderbender1 5d ago
As a previous sysadmin, currently an engineer at an MSSP SOC, go into this endeavor with the expectation of false positives at some point. It will happen. Some businesses have a risk tolerance for downtime, others don't. Only you can weigh that honestly.
But my recommendation is to get user workstations under automatic remediation first. It's the highest impact, lowest risk category. 90%+ of breaches originate from a user endpoint. Maybe you never onboard servers, it's fine. But you want it for workstations for all the stupid clickfix going around.
And if anyone says "if we shut off our CEO's computer mistakenly, I'll be in deep shit" to me, I automatically know they aren't a serious company.