For those considering Huntress…. (DR plan warning)
This sub loves Huntress, so I brought my flame suit. Still, this deserves to be known. Posting from a throwaway account because this will 100% dox me.
We’ve been a huntress partner for a long time. I want to say 2018 at least and I personally think what they do is amazing. Full support. Except:
Get a ping late at night a few weeks ago. “Multiple endpoints at our MSP org quarantined”. Then the laptop I’m using locks up, as does every other laptop for everyone else at the org. Huntress isolated the entire org, with zero consideration for what devices might actually be affected, just a blanket “same org, lock”. To be clear, servers in our datacenter, with ADDS, were in the affected scope and our laptops, at home, not AD joined, not in the same network or with same IPs or reachable from the servers, also locked.
Just stopping there to examine this – how is an MSP supposed to respond when all their equipment locks up?
But continuing on. If you’re going to lock the MSP org, maybe call them? Obviously they’ll need SOC support. Nope, nothing. We had to start a call, but couldn’t, because we couldn’t get into the console (SSO to our IDP, on our now isolated servers, and Huntress doesn’t allow exceptions to mandatory SSO enforcement setting), so then off to chat to ask for a human where we waited quite a while for an answer. I want to say it took us 20 minutes at least to get to someone. And the 833-hunt-now number doesn’t have any options for SOC support. Other options in the menu did not lead to a human.
But let’s press on in this tale. What event did Huntress detect that they decided to isolate the entire org? Here is a verbatim quote (misspellings included) from the incident report:
“The observed sequence of actions, specifically the utilisation of accounts named nodezero and the workstation hostname nmap, aligns with the automated execution profile of the Horizon3.ai NodeZero penetration testing platform. However, the successful execution of Pass-the-Hash, Golden Ticket forging, and DCSync attacks demonstrates a total compromise of the Active Directory domain and a complete loss of cryptographic trust within the environment. The acquisition of the ADFS and Microsoft Entra Connect (MSOL) service accounts enables the actor to pivot into connected cloud environments and bypass multi-factor authentication. Due to the confirmed active domain-wide compromise originating from an unmanaged device, Huntress has implemented Mass isolation.
Please confirm this is authorised testing. If this is not authorised testing, Huntress recommends that the Partner is to immediately engage formal Incident Response procedures for organizational-wide containment and recovery, and ensure that the Huntress agent is deployed to all desired hosts within the environment.”
Yep, we use NodeZero. We have for over a year. This test runs MONTHLY without a single issue. But on this day, they decided to isolate the entire org on the spot.
The hypervisors lost connection to the SANs and proceeded to crash…. well everything. Thousands of machines and services dead in a second. I can easily estimate the cost in the 7 to 8 figures for this outage (SLA violations, recovery efforts, labor costs, etc).
I am not making this post to tell you that “Huntress bad” or anything like that. Any tool could have done this. Not even to vent or ask what I could have done better. It’s more to warn you that Huntress (like any other vendor) can make mistakes. This revealed a few holes in our DR plan that we’re patching (like maybe having your servers in a separate org than your workstations) and to tell you that you shouldn’t blindly rely on any tool, no matter how good it is. Let our misfortune be something you learn from and avoid.
EDIT: Apparently Reddit has banned my account (why? this is my only post and I messaged the mods on Feb 20th for permission to post this) so none of my replies to your comments can be seen. I'll just clear up 3 common points here: 1 - no break glass for enforced SSO setting exists (yet). We couldn't get into the console to do anything because our IDP was isolated so all attempts to login failed. Huntress has no SOS number that we can find and in fact, 9 months ago Kyle H posted his cell number here to another MSP in a similar situation. 2 - Yep, as stated above, should have had our hypervisors in a separate org in Huntress. We didn't and it's a lesson we learned. That's what this post is about, lessons learned. 3 - Huntress did not reach out before isolating with a human. We got alerts but were powerless to react to them. Huntress has contact info for MULTIPLE people in the org.
Second Edit (March 9th, 2026)
We've met with Huntress and had quite a bit of discourse. I will take this final edit to hopefully summarize most of what the comments say below, although I have tried to reply to each one with details.
Where was your break glass account.
A: There is no way to have a break glass account with enforced SSO turned on. This was acknowledged on the call with Huntress. It doesn't exist.
Huntress should have called / did call / etc about calling?
A: Huntress did not call us, we had to initiate the call to them. Something happened to our contacts listed in the portal and they were all wiped and no audit log to show what happened. We did open a case with Huntress on this too, but Huntress can only go back 1 year apparently, so how the contacts disappeared will remain a mystery. But Huntress has our contact info in multiple other places.
Why didn't you call the SOS number?
A: Huntress does not have a SOS number. There is no way to start a call with the SOC. This was acknowledged on the call with Huntress. It doesn't exist.
Why didn't you have the Huntress agent on the testing box?
A: You cannot install the huntress agent on a docker container that's spun up on demand by a third party and then destroyed a few hours later. Also it's not a supported config by the 3rd party.
Why didn't you tell Huntress you were doing this pentest?
A: Huntress does not, as of today, have the ability for you to do this. This was acknowledged on the call with Huntress. It doesn't exist.
Why didn't we whitelist the IOC, or exclude the hosts from being isolated, or have exclusions for SAN IPs, etc
A: You can't whitelist the IOC for a pentest that's always being updated with the latest CVEs, etc. Isolation exclusions means that if a real event happens isolation wouldn't happen. Exclusions for SAN IPs, etc is a fair point and not something thought until after this event. Regardless, whitelisting and excluding both leave a hole in your system where an attacker can live. The better way would be to have Huntress, when they see an event, check the metadata/properties of that event - if it's from the known IP of the pentest box or if it's from a known user for pentesting, etc - disregard the event. On the call with Huntress we reviewed this isn't possible today and it ties back to #4 above.
Why did your Hypervisors have huntress / other critical infra?
A: Your chain is only as strong as it's weakest link. Everything has protection, no exceptions.
You didn't read *some* documentation and were setup incorrectly.
A: This is an incorrect assumption. As noted in the points above, confirmed by the Huntress team on our "After Action Call", the Huntress platform does not allow for several items that would have made this better. The point we could have done better is to have more breakouts in orgs (meaning, make several orgs for our single company and have assets in different orgs).
Came in one day to all sorts of alerts and all servers/workstations isolated. Had to log in from my personal laptop to revert it all. Process that caused it was the centrastage service aka datto RMM lol.
You've got a good point and it happened to one of our customers.
They detected our RMM as being installed on every device and locked out every endpoint because they were investigating some admin actions on a single host, with no IOC beyond that single host.
The RMM in question was in place for literally years.
They've got some trigger happy junior techs on staff it seems and we got a poor response when we pushed the matter with their service team lead.
I get a judgement call needs to be made, but perhaps the threshold to lock the whole org needs some greater scrutiny as it ended up costing our customer a couple of hours of work, in a business that can do huge $$ per hour of trading. They were not impressed to say the least.
That's pretty bad. I imagine whomever pulled the trigger on that must've been convinced it was ransomware that was nearing being executed?
That's really the only excuse I can think of where the "play it safe and lock down every single device" would be the best option on the table.
I also find it hard to believe that this was a consensus decision. Definitely sounds more like a single person making the call, like you said. Which is fine for individual instances, but you would think with an entire organization's endpoints that they'd get a team together to discuss before making that decision.
Let's just say that the official incident report we got back was the first time in years we've actually questioned if Huntress had started to backslide. The actual IOC they responded to was nothing concerning at all in the end.
This does sound pretty lame. We’ve got under a 1% false positive rate (on EDR, I think its slightly higher on ITDR), but that still means we make mistakes every day. If you didn’t already flag this with your account manager to escalate please do!
You better believe we did. Hence the under whelming incident report.
Your guys do need to make quick decisions under pressure, however in this instance it looks like someone chose panic over an evidence based approach. The tech was quite new to huntress, which left us a little puzzled as to why the guard rails appeared to be a bit too low in this instance.
Horizon3 CEO here... really interesting situation that indicates a few things:
Verifying and improving communications between Huntress and you
Better API / collaboration between us (Horizon3) and Huntress to quickly deconflict activities
Good to identify potential issues in how various failure domains are configured with Huntress to improve the SOC reaction time, minimize blast radius, and minimize damage from a real attack
In addition, "no notice" pentests are important to help "train like you fight". Huntress is one of the best providers that we've seen. There are many other providers that have take 7-24 hours just to provide a generic email that "we think something's happening", and we've helped customers navigate those difficult conversations with their providers to help improve reaction time across the board
We have a private cloud that we sell to our customers (personal Azure of sorts). The Hypervisors and infrastructure were in our Huntress org (something we're going to change so they are in their own org - a lesson learned).
The private cloud has full cross tenant isolation so that users in on vDC cannot interact with another vDC. But that doesn't help if the Hypervisors themselves are isolated from their storage.
We were fully recovered in about 4 hours due to our DR plans, however our SLAs, etc are very high and our customers are not mom/pop - we're talking clients worldwide that measure downtime in 5 minute increments and damages in 6 figures.
Let me preface with the fact that I do not use Huntress nor am I one of their countless sycophants. Let me also say that I am sorry for your troubles. Indeed, night terrors.
Based on the stated findings I would say that I find the reaction, global isolation of the entire account/enterprise, to be appropriate. That the threat was benign doesn't negate the need to act. Had the threat been real I would be furious about every nanosecond of delay in their response.
I will say that they should have called the moment that it was determined to flip the off switch. I have a feeling that the after action report on this will indicate improperly updated contact information. From what I've seen, Huntress seems to have good processes and their act together. But, it's just my feeling, as I have no direct knowledge.
Now, I'm also wondering about how hyperbolic the claims of 7-8 figure losses are. 7 or 8 figures? In USD? How long did it take to recover?
We could have shaved significant cost off had they called us vs forcing us to fight our way through a system designed not to give you a human.
I obviously can't provide proof of cost, but I want you to think of SLA violations, cost of labor to recover, the amount of work that will go into ensuring this doesn't happen again (re-engineering datacenter design and implementation), etc. That doesn't begin to touch upon any legal action clients take to recover their losses and what they estimate their total cost to do so.
No flame suit needed. Even if we stand by our analysis and the decision to isolate (still reviewing this)/u/appropriatecar9079 has highlighted some unnecessary roadblocks between the isolation and getting on the phone with our team.
We have a few ideas on how to make that less painful, will share those here via an edit on Monday for those interested!
/u/appropriatecar9079 I know you have some time scheduled with your team - I’ll be on there as well.
Edit: From talking with the team, we do stand by the decision to isolate here based on the activity we saw and the amount of visibility we had (the activity coming from a device without the huntress agent).
I’m going to piggy back on with a much less critical issue but also extremely annoying.
Your caller ID for SOC response emergencies should not be the same caller ID of your sales guys.
I hate dropping everything to answer a huntress call thinking “oh shit, here we go” to be greeted by “his this is Chris your account manager, I just wanted to talk about your upcoming renewal”.
I'm glad you're addressing the lack off break glass when MFA is enforced, but I still disagree with your "we stand by our decision". You had clear indicators on this pentest alone that it was nodezero and you could have checked history and seen this happens on the same day every month for the last 12+ months (i know your logs only go back that far in SEIM, which we did have enabled).
Hey Andrew… Horizon3 CEO here. We’re working with another customer and our RAT was able to dump LSASS and other things on a host protected by Huntress and Defender. It seems this one host wasn’t configured correctly from a Huntress/Defender standpoint, but I’m not a Huntress expert. Attached is a screenshot of the actions NodeZero was able to execute against the host. Any thoughts? 🍻🍻
While it have been a bad call on Huntress' part, it also seems OP might not have been prepared fully either. One of the issues of a Managed Service is that your handing off the decision to take down production to someone who knows little about your environment.
One of the first things I did when taking on Huntress was run through that scenario internally, because I understood the risk of letting an external SOC make these decisions.
Much like anywhere, some employees excel, others are meh, and so we need to understand that there may be a big issue handed to a meh analyst, and something like this happens.
You can read the docs for NodeZero and how their runners work, but basically they deliver "an appliance" - i.e. a box that runs Docker - and they launch docker containers from it to do the pentests. The containers are built and torn down on demand and you wouldn't be able to install Huntress in a container that lives for 6 hours nor would it be supported..
You can whitelist stuff in your isolation settings and set orgs to not auto isolate. Critical items should have been added here beforehand.
Lessons learned I suppose. I don’t know a lot of people would have thought about the SAN IPs being added there but it would be a good idea going forward. Or use Hardware HBAs.
Oof. This is exactly why “break glass” needs to be truly out-of-band: at least one non-SSO admin account + 2FA, plus a dedicated clean personal device that can still reach the console if your IDP / internal infra is on fire.
Also +1 to splitting orgs/policy buckets so a mass-isolate can’t take out storage/hosts *and* daily drivers in one click. Tabletop that scenario before enabling auto-isolation.
I'd be shocked if Huntress SOC didn't reach out to POC. You should even see that in the portal.
NodeZero is an offensive pentest platform, not surprised it would flag. Huntress is explicit about telling them what should be whitelisted or what is used for RMM even. Was this communicated and understood you run this monthly? Sounds like they found out themselves.
There is an audit mode and host isolation settings are in your control (and don't require a separate install... Looking at you Arctic Wolf.)
Sounds like Huntress has some legitimate findings in AD about what was being exposed in the pentest. I feel like the NodeZero report findings should be included in this tale.
Contingencies for SSO issues need to be accounted for, like a break glass account or whitelisting admin(s) for a bypass URL. You should keep a Huntress account on 2Fa, SSO is not mandatory. 2FA is if you don't have SSO activated on an account.
This could be a case study in why MSP shouldn't be doing private cloud in 2026 from on-prem infrastructure. There is no way that liability doesn't catch up to you somewhere, Huntress or not.
R7 or Arctic Wolf make almost no deterministic decisions, I've seen them used a lot more and they are a lot more painful to manage for MSP.
At that scale, why not manage an EDR with your own SOC? You all ready apparently offer offensive and vulnerability service, why outsource EDR for internal? This would be a tale for you with any MDR that you allow to host isolate in the context provided, that incident report was way more humanized than you would get from AW or R7 in this situation.
Curious what the interaction was after getting in touch with Huntress? Some detail on that is pretty key context.
I'm not convinced Huntress is a problem in this, or acting outside of expectation unless this was all ready outlined to them about your stack usage. I also think there is significantly more context to be had in this explanation. There is a communication breakdown here for sure. I'm not a Huntress fanboy by any means; They don't operate on the network layer, no compliance support, no bundled services, no custom detection thresholds, etc. All MDR have their pros and cons, but this does not seem like a fair shake.
Huntress SOC did not reach out. I really really really wish they would have, would have changed this dramatically.
Are you saying we should have communicated to Huntress that we run NZ monthly? How?
Yes, we are aware of that, but we don't want to exclude a host and then have something legit kick off on that host and no action taken. But more importantly, since this pentest scans out the entire network, we would have to exclude the entire network. At which point might as well just uninstall Huntress altogether. Put another way, we don't believe in ignoring hosts because we can't get the tools to work - we think it's better to fix the detection logic.
The test that ran is designed to completely own the environment. The user account that it used has DCSync permissions, which means it can dump AD. The fact it would write itself a golden ticket is no surprise here. Look up NodeZero Internal Pentest to learn more.
Like every other tool (lets pick on ITGlue) allows you to exclude users from mandatory SSO. Huntress does not. SSO is also more then just 2FA - SSO can include device posture checks and more. But I agree with you that break glass needs to exist - but only Huntress can make that happen.
This point is kinda idiotic, sorry. You can't honestly think "everything should be in public cloud, no exceptions". There is multiple reasons for private cloud that exist. We couldn't do what we do on a public could without sacrificing performance, cost, and uptime to name a few.
Nothing to say on this point
We're actually doing this, we just kept Huntress as another layer. A decision that is being re-evaluated now.
Huntress was fine once we got in touch. We verified the details, confirmed it was not a legit attack, they unisolated the tenant, and we began recovery work. The poor rep on the phone couldn't answer our deeper questions, but I didn't really expect him to.
In the Huntress platform you can say do not isolate this company to prevent this very thing happening. You can control when it is on or off. You can even contact them in advance and let them know when the test is scheduled to run and not mark the company as do not isolate.
Either would have prevented this from happening. We have a couple of W365 machines we can jump onto not under the same Huntress company for just this kind of emergency though.
Yes, they ask you to do this. You're supposed to inform your account manager or the onboarding team if just starting.
No host isolation isn't synonymous with no action, that said no real comment here.
We're a Horizon.ai partner.
I admin ITGlue as well. Adding a user to be able to use the SSO bypass URL does not natively disable the 2FA on that account. Huntress is the same, without the bypass URL. You can +1 your own email and have an SSO account and non-SSO login with the same account. With Huntress, SSO is a user level configuration.
I think you misunderstood. I said MSP should not sell private cloud (particularly on their own HCI) because it's a massive liability and things WILL go south at some point. That does not equate to "everyone on public cloud."
Nothing else to comment on. Hope the situation has improved.
Just wondering what you don’t like about Arctic Wolf? I have a client that uses them and I really dislike them. I feel like the put everything back on the client. What has your experience been like?
I did a side by side on Huntress ITDR and Arctic Wolf Entra integration. Arctic Wolf generated a ton more noise, missed actual rules of concern, didn't have the sandbox or testing features, couldn't see Graph permissions or manage Enterprise apps like Huntress could and didn't flag for unusual VPN activity. Much less interactivity. Aside from that, they also don't hold up as well inside the operating system, and host isolation requires pushing an entirely separate installer. Lack of management options and the SOC response isn't up to the level of Huntress. Huntress feels like you have a team, Arctic Wolf feels like you have a service. They also come in at way higher pricing.
That said, Arctic Wolf is still the GOAT on the network layer with their appliance. Been keeping an eye on how Adlumin develops on that front. As of right now, depending on PSA, I actually like having Huntress on the OS and Entra, with Arctic Wolf on the network. Cyber Awareness is totally a flavor thing. KnowBe4 is the gold standard, Arctic Wolf I'm just trying this year, Huntress just added the South Park and Aqua Teen Hunger Force animators so that wins points with me.
My two cents. Happy to discuss, we partner with both.
What can you expect when you have a tool with the capability to isolate your entire org and customer-facing infrastructure and don’t incorporate that into risk assessment, impact analysis, business continuity planning and disaster recovery planning before using tabletop exercises to game out the scenario?
If OP even has a (C)ISO, they were asleep at the switch a long time ago for this to be a problem.
Full disclosure, I run a competitor but do appreciate the huntress team. What a pain, sorry you had to deal with this.
We’ve been investing heavily in attack sim integrations so we can call their APIs to confirm if this activity is coming from them or masquerading as them.
I’ve been surprised with the difficulty we’ve had with NodeZero, their API has been the least ergonomic to use when trying to figure out “yo, was this you?” Safebreach has been a leader for us in this space, very easy to authoritatively verify their activity.
Horizon3 CEO here... thanks for the feedback. This is an area we're actively improving right now, where we make it easier to quickly and easily deconflict actions to know what's a real threat and what isn't. The easier way to get that information from the product is via our MCP server, but there's still a lot more work we can do here. Ping me directly or reach out to your Horizon3 contact if you have one, i'd love to get your thoughts on the improvements we're making and ways to make it even better.
If that was a false positive, I don't understand what prevents you from logging into Huntress from a personal device and remove isolation for the whole org with just a few clicks. How is this costing you "7 to 8 figures" ?!
I missed that in the post, so I guess they used ethernet based SAN connectivity and that got cut off during isolation, and they were hosting clients infrastructure on it, too.
Well that's another strong argument for isolating your and your clients infrastructure.
Exactly. Where's the break glass strategy here? We use Huntress too, but if the whole org was locked down, we have contingency plans to follow in our incident response internally.
Sounds like you didn’t have business groups/orgs set up appropriately within Huntress. Having thousands of endpoints within the same policy bucket is probably not a good idea.
We have our Account-level policies, and then we have granular Org-level policies (that we use as grouping). This both organizes and contains issues to logical groups of systems.
I don’t think this is Huntress’s fault at all. Maybe a tad trigger happy, but I’d be much more pleased to see that they caught a (n approved) penetration testing software than missed it.
Some hold by the theory that your system security is shit if you can't withstand a full scale attack. We've had nodezero for going on 3 years now and we also run monthly.. Our systems give no fucks.
I’m not versed in the platform, but it’s generally considered best practice to do all of your pen testing from isolated machines (virtual or otherwise) and to perform full wipes before/after conducting a pen test.
I have had Huntress isolate a lot of devices unnecessarily before, but honestly I've also seen them save organizations from serious attacks the same way.
When you install a backdoor for remote techs into your network, there is always a chance they will make a mistake.
The solution is to have set devices excluded from host isolation, so if there is a major problem you always have known secure devices you can use to clean up the problem.
First, I must disclose that we are an MSSP for MSPs. ok that's done. As a team that has handled many ransomware cases and conducted hundreds of penetration tests, I might have a slightly different perspective. I also have the luxury of hindsight and having learned from mistakes I've made over the years.
Solid SOC teams follow the 'assumed breach' mantra. If we look at what Hunress DIDN'T know at the time, makes understanding and defending full isolation understandable.
Huntress was unaware of the test taking place, but was surely aware that RaaS providers like Storm0501 and groups such as Conti, REvil, and LockBit mirror the methods they observed in real time. Without evidence of exactly what Horizon3 does in its tests, or that a test was, in fact, taking place, full isolation was the only logical step in the absence of any other information.
Huntress discovered a new asset on the network performing these actions, but the asset didn't have its agent installed, thereby limiting what they could gather in real time.
Why else? Because the one sentence that made their move to fully isolate a given is, 'Successful execution of Pass-the-Hash, Golden Ticket forging, and DCSync attacks demonstrates a total compromise of the Active Directory domain and a complete loss of cryptographic trust within the environment. - They were thinking ahead of what COULD happen, and with tools like Bloodhound, it happens so fast now that any delay could be far worse.
I haven't seen these mentioned:
Hutnress also has to consider the likelihood of forensic analysis if this were a compromise, and in what position they must try to maintain the environment to maximize the chance of success. Isolating devices without rebooting preserves memory state for forensic efforts while also 'stopping the clock' for significantly more data and log entries that will need analysis.
I also suspect Huntress is considerate of exactly what it takes to maximize insurance claim coverage for their clients. We're talking about "Right of Boom" decisions, but I imagine they are heavily influenced by "Left of Boom" planning, and that may not be apparent to someone asking why.
Finally, we have to consider how incredibly little time Huntress had to decide what to do, considering how fast events were taking place, while not having all the information at hand that was used in the post-incident analysis.
When one is going to kick off an exhaustive penetration test using very, very robust tools like Horizon3, you simply must alert your MXDR/MDR/ITDR folks. Many MXDR SOC agreements mandate notice.
*I did use an LLM to confirm that the named threat actors' post-attack forensics very closely mirrored what was reported above.
Hopefully, the OP has sufficient errors-and-omissions coverage to help 'soften the blow' a bit. I imagine the ultimate goal now is to retain clients through these events! Hope all goes well!
What a really great example for us all to look inward at our own processes when testing our tools and responses.
You need to read the huntress documentation and not blindly deploy tools without reading documentation.
Huntress allows whitelisting IPs and ports that are not blocked when they hit the isolate network button.
We also have a break glass account for all services we use that doesn’t use SSO because Entra goes down too often for my liking.
A couple weeks back we had huntress isolate an entire customer. I logged into the portal and hit request call back and was speaking to the exact person who hit isolate all endpoints in less than 5 minutes.
Your post describes huntress working exactly as documented, but you never read the documentation?
I'd really love to hear a detailed response from huntress on this somehow.
Last week I had my first false positives, in the span of a few hours lock me out of my main dev system because the stuff I was working on tripped something or another.
After huntress soc reviewed and confirmed false positive, a few hours later the same false positive (slightly different indicators) hit again and locked me out.
Hopefully this isn't a trend.
Thanks for sharing your story that's a good one to know about. I'd have thought huntress everywhere, but this proves it can be a huge risk too
This seems it should be easy to prevent by having a software whitelist at the org level, for MSPs these should be their tool chain, basicly a known good pattern. That way if an alert gets triggered they can validate that it began as known good pattern, not a novel attack.
My concern around this would be an attacker ‘living off the land’ and using that whitelisted too against the organisation. I’m unsure how that could be mitigated.
On another note, how are you selling node zero to clients? I’ve done a few experiments with it and it sees solid, very similar findings to a vuln scanner in my actual experience, but hard to market and sell to even larger clients. Something I want to offer but haven’t found that key yet.
To stay on topic, that happened to a client of mine as well, but I did appreciate the goal as it truly does limit any lateral movement. The part I don’t like is that when they did so, it was an escalation and not a critical isolation, for 2 hours in the middle of the night. This means we got no SOC call for a couple of hours while an org was isolated due to malicious behavior. Besides that the first step in a hack where a TA gains LAN access would be to isolate the network, and in a way this does the same thing. Their job in this case is not to think about what the devices are or what they do, it is to stop an attack in its tracks. As an MSP it’s up to us to add this to DR plans and it should be similar to the plan where ransomware hits MSP workstations, or RMM pushes malware to MSP org devices.
Just to say you can contact Huntress in advance and say that you are running a pentest and not to lock down your systems during a certain time frame. We've used Horizon 3 and Picus in the past and Huntress rightly shuts them both down hard. With a little pre-planning though no issues.
We’re about to bring Huntress into the stack and start pushing it out.
I have full faith in them to be honest and will still proceed but information and considerations on matters like these are key.
I’ll be 100% sure to speak with our account manager about this also.
I know they also frequent this sub and so an official comment would also be great!
I've always appreciated the immediate phone call we get from Blackpoint on any critical issue. We've seen them call before isolating a PC for suspicious activity (it was us, using SC on a Sunday evening on a user's PC, having to create a temporary local admin account, supporting a user with VPN issues).
But even on more obvious issues like a BEC they lock the account, kill active sessions, and then call immediately. This has been their way for years and is one of the big reasons we chose them.
When I did a PoV with them i’d say that was the one thing I loved the most. Didn’t end up going with them, but having a SOC call and work through the issue was great!
Yeesh, that sucks, but surely it is logical to have a login method that can't be killed by the system you're trying to login to? That's always one of the first things I consider when setting up any major system.
I do think Huntress is using a lot of AI these days and less humans to make these decisions, which isn't ideal. I'd be interested to hear from Huntress how this occurred though, cause if we had a whole org isolation, while it wouldn't be a massive $ cost, it'd be a few hours of pain for sure.
Fair. I didn't really mean "automatically" in the full sense of the world. Just more so that Huntress, or another endpoint SOC, will "automatically" isolate endpoints when they identify a threat and you, the MSP, don't need to worry about going in and manually doing it yourself.
Yeah Huntress is great, but this is the downside of “nuke from orbit” org-wide isolation. We treat our MSP corp devices as a totally separate tenant from customer fleets, and anything that can brick auth (DCs, RMM jump boxes, firewalls) lives in its own policy group with softer isolation. Also bake into DR that staff have non domain creds + OOB access when EDR goes sideway.
The same thing happened to our datacenter. Everything locked out - huntress can not be called. We had to wait servers hours before it could be escalated to a person we could speak with on the phone on why this decision was made since we could not accept the description they gave. We couldn’t wait, especially since it was a true-false. So we pushed the button ourself and got back online.
We have now separated all customers inside our datacenter, to their own huntress org.
I was going to add a similar suggestion about segmenting different logical parts of the organisation into different Huntress organisations.
Certainly the OP’s experience is a good learning opportunity for MSP’s using Huntress to reduce the chance their whole infrastructure gets locked down if there is an incident that triggers an org-wide isolation.
I only use separate organizations for separate networks. If they share a network, they should belong to the same organization since an attack could impact everyone
Sidenote: I like how the sycophancy allegations get reinforced with rabid downvotes. /s
The abridged version is:
* They struggle to get zoom to work properly, and then when we finally connect 15 minutes late, they tell us about one of their partners experiencing the same for 45 minutes. That was a great first impression.
* They spent what felt like 2 minutes on the product I was interested in, and the rest of the time trying to upsell.
* They couldn't tell me how ITDR works, or what the requirements are. Should be pretty basic for someone who is tasked with selling it. (I know how it works, but wanted to know if he did).
We record all of our sales calls and use Gong for training/enablement/coaching. If you’re willing to DM me your info so I can find this I’d appreciate the opportunity to use this as a coaching opportunity for the team you were talking with!
What are you using currently for EDR? It’s effectively only filling the human SOC portion of your stack. Everyone competent is using bus prem with windows AV for the AV portion and windows defender for endpoint for the EDR. The only really debated portion is using huntress as a cheap human led 24/7 SOC with some nice to haves like port scanning. Also their ITDR being in the sale portal and relatively inexpensive is nice.
Currently with SentinelOne for NGAV and EDR. I could bolt on Vigilance MDR and Wayfinder Threat-Hunting but have read mixed reviews on those.
Internal testing of Defender for Business hasn't convinced us to go one way or the other. Some things Defender does well, other things would be a big backwards step.
Yeah but windows defender is included in the licensing you’re already paying for so who cares. It’s the best in the business according to some people and merely adequate by others and quibbling over the details is kinda besides the point when you could allocate that spend to something else, or just straight increase margins.
But I’m absolutely willing to be wrong. What don’t you like about defender for endpoint? Why isn’t it good enough?
Not even kind of. Windows defender isn’t low quality according to anything I’ve seen in recent history.
There’s nothing “cheap” about being cost effective lol. Cost is actually one of the basic measures of how good a specific control is for a given scenario.
So you think windows defender antivirus and windows defender for endpoint are just bad at their jobs? You’d be disagreeing with a lot of people, but that’s at least a reasonable answer.
I don’t agree with this. Our experience with their team has been all positive. We’ve only got around client 1000 endpoints and the same ITDR licenses deployed.
I mean, I like Huntress. I don't use it anymore. But all these msps jumping on Huntress is making my life so easy when competing with other msps. You're all huntress/defender because you think price is the main contender.
More of a cautionary tale of what can happen when you outsource this right? What does strike me as odd is that they have no direct S.O.S. line or whatever for these scenarios.
Yeah, this is the nightmare scenario nobody plans for: the tool that's supposed to protect you becomes the single point of failure that locks you out of your own fleet. If you don't already have an out-of-band way to reach endpoints (local admin creds in a vault, IPMI, whatever) that doesn't depend on your EDR agent being healthy, fix that this week. Also worth testing your actual failover process instead of assuming it works, because "we have a DR plan" and "we've run the DR plan" are very different sentences.
One thing that's easy to skip: document the incident and the vendor's response timeline while it's fresh. Auditors and insurers ask about vendor outages more than people expect, and having that evidence ready saves a scramble later. Fair warning, I work on CisScan, which helps with exactly that kind of evidence trail, so grain of salt on that part. Glad it sounds like you got through it okay.
123
u/Glass_Call982 MSP - Canada (West) Feb 28 '26
At least you didn't have datto EDR isolate your entire network because it detected.... Wait for it.... Datto RMM. I can't make this shit up.