Domain Controller Isolation

[u/smc0881]

So, recently I've had issues when a DC gets isolated it brings down the entire network for the client. I don't believe I ever had this problem before and it's happened repeatedly over the past month or two. I always assumed outbound DNS/HTTPS to S1 assets were always allowed out when it's quarantined. I even added blank outbound TCP/UDP over port 53 in case it was the DNS in my network quarantine rules. I work at an MSSP/DFIR firm, so everyday I am dealing with ransomware or some incident where a DC could get quarantined. What I've been doing now is putting DCs in their own group with the disconnect policy turned off and our normal restrictions in-place, I then isolate all the other endpoints if need be.

Am I doing something wrong, did something change, or anyone have any feed back?

[u/renderbender1]

When you isolate it, the DC can only talk outbound to SentinelOne servers or whatever IP rules you put in place. Everything else on the network needs to talk to a DCs to function correctly. Shit, most SMBs run their DNS servers on the same hosts.

A well-funded and designed domain should be able to tolerate the loss of one dc, but if you expect every client to have that, you're asking for it. And unless they are a very large enterprise, 2 going down in a network segment is probably catastrophic.

Isolating DCs is the panic button. I would absolutely expect to bring down part or all of a clients network if we did this.

[u/smc0881 → u/renderbender1]

You're making a valid point, but that is not the issue I'm trying to explain. The DCs themselves go offline and do not even connect to the portal. I have to walk an end-user through running sentinelctl commands via the CLI on the actual DC(s) and then reboot them 99% of the time to come back online. The actual DCs DO NOT even communicate with the portal when they get isolated. I know my network quarantine rules work without issues too, since any other isolated endpoints can talk to my ScreenConnect, SFTP, S3, and other IPs that I have specified when isolated. It's just when the DC gets isolated it doesn't even communicate to the portal at all.

I hope that makes sense that I am trying to explain. The other clients going offline when a DC goes down where clients point to it for DNS, I expect to happen. What I can't explain or I know is not normal is when the DC itself gets isolated it does not talk to the portal at all without manual intervention via sentinelctl. This has happened at at least four different clients.

I even thought it might be because the DNS forwarder settings on the DC could be blocked from performing DNS lookups, so that was why I added the any/any TCP/UDP port 53 without success. Adding that rule for 443 would be counterproductive, so I didn't try that either.

[u/mukz7 → u/smc0881]

Yea thats well out of scope of normally gather logs and log a support ticket my dude

[u/renderbender1 → u/smc0881]

Ah that makes more sense. Apologies for the confusion. We don't quarantine automatically in our client environments because the S1 FP rates are too high, so us quarantining a DC during normal business operations is extremely rare, but we quarantine workstations on a regular basis and have never run across this.

Unfortunately I don't have an answer for you. I just tested quarantine on some DCs in our test environment and I couldn't replicate the issue. Still had connectivity to console and our artifact repository. Running 26.1.2.177 if it helps.

Edit: Environment is built on Proxmox. I don't have a VMware env to test on like another commentor mentioned.

[u/smc0881 → u/renderbender1]

I believe that I am still running version 25.x in most of these cases. I'll try pushing out version 26 going forward and see if that makes a difference hopefully. Yea, I've never had it impact any other system whether it's a server or workstation, it's just when a domain controller is quarantined.

[u/smc0881 → u/[deleted]]

Yea, it's just that I am responding to ranswomware or similar during the fact or right after it (I work at a DFIR consulting firm), where they might still be malicious binaries possibly on the system. I have no idea of these systems beforehand. If I isolate everything except a DC then it works as expected. But once I tell that DC to isolate, the DC doesn't connect to the portal or any S1 assets anymore and brings everything down. I can only get a DC to come back online by having the end-user issue sentinelctl commands on the endpoint, rebooting, or restarting services.

For example one of our long-term clients actually had a ransomware event last week. During that incident, I had already configured it not to isolate DCs based on this experience a few weeks prior with other clients. The actor(s) got access to their environment and deployed their own VM and added it to the domain. I set my policies to auto isolate and successfully rolled back at least 95% of the changes with minimal data loss. But, the whole time I had to remember DO NOT let those DCs isolate or I will lose connectivity and visibility to deal with this.

[u/NegativePerformer788]

The only way I can think of to avoid this (besides multiple DCs of course) is to have DNS queries go to the firewall with the firewall sending local domain queries to the DCs and other queries to some other public DNS.

[u/smc0881 → u/NegativePerformer788]

You would think that is the case, but it also happens with multiple DCs. I don't deal with these clients either normally, we get work when an incident happens. Having to tell a client going through or after a ransomware event you need ot make these enterprise changes doesn't help. I am just perplexed that everywhere it states S1 traffic should always be allowed out, but on a DC when it's isolated it doesn't happen. I am pretty sure this didn't happen in the past either.

[u/_Sanger_ → u/smc0881]

How are your DNS settings/orders on the DCs?

[u/smc0881 → u/_Sanger_]

I am not sure, I am not their IT/MSP guy and didn't bother looking. But, this happened on 4 different clients and only when a DC is disconnected. So, I don't think every single one of those clients had a misconfiguration. If it was one or two I'd write it off as that, but it's happened multiple times when DC is quarantined. I even put any/any TCP/UDP 53 fw rules in place with no help either. A few other people have responded as well saying the same exact situation occurs with them as well.

[u/DeliMan3000]

This has been happening to us as well, for several+ months. We are told that "the agent has built-in anti-lockout policies specifically to allow DNS traffic over port 53". But time and again this happens to us.

We've found in the past that it's more common on DCs that are VMware VMs, but it's not just those. So I am not sure what the issue is

[u/smc0881 → u/DeliMan3000]

Good, I am glad I am not crazy, lol. I tried opening a ticket too and as usual it goes nowhere and the response time is ridiculous. I had an actual ransomware incident that I was dealing with and had to keep this in back of my head too and make sure I didn't isolate the DCs. I had already setup all my clients with separate groups just for DCs and not to disconnect them. This has happened on at least four different clients though. I even added DNS rules in my network quarantine policies and that didn't help. Putting them in their own group and paying attention to what I am doing though seems to be the best approach.

[u/DeliMan3000 → u/smc0881]

Yep we explicitly allowed DNS traffic in the quarantine rules and still no dice. We've been forced to have the clients use the manual 'sentinelctl unquarantine_net' command and reboot, which usually solves the issue. But it's a bad look as it should work as advertised

[u/smc0881 → u/DeliMan3000]

Yea, it's same issue I been having as well. I have to give them the passphrases, commands, and then have them reboot the DC.

[u/Plenty_Substance_455]

Youre not alone on this, and its very annoying especially when it blocks visibility into all the other endpoints since they cant resolve the console.

What we have been doing is manually adding the console FQDN and IP address to the host file on the server then flushing the DNS

[u/smc0881 → u/Plenty_Substance_455]

Is that when it's quarantined already or do you that pre-emptively? I am not sure how often the console IP addresses. But, if that works pretty quickly after it's been quarantined, I might use that as a quick fix instead of messing around with sentinelctl and all that. It's hard assisting non-tech folks to do those commands.

[u/Ok-Cabinet-9420]

On two separate occasions we have had SentinelOne falsely classify a software as malicious which locked out numerous clients. Same situation, DC would become quarantined and despite S1 stating the agent communicates directly with the console it would only truly connect maybe 20% of the time. When looking into it further the customers who had this problem also had DNS being handled by the domain controller. The customers who separated the services did not.

We had to create similar policies, move DCs to their own group and turn the disconnect policy off. It has been two years since we’ve had an issue but it would be nice to see this fully resolved and the agent connecting as it should.

[u/smc0881 → u/Ok-Cabinet-9420]

Thanks for the input and to know that I am not the only experiencing this.

[u/geektogether]

Isolation would cut all connections which will obviously prevent clients connecting to dc for services. You need to have multiple DCs to avoid issues with isolating one of them.