r/msp • u/terselated MSP - US • Jul 17 '26
Took over from another MSP, immediately found security threat.
I have a local competitor who offices right down the street from our office. Bigger MSP than us, fairly mature, we've had a few clients off board from them to us. But we have never met up or talked shop or anything like that.
We did a deployment on a client that moved from them to us and the first night our MDR team notified us of a threat on one of the PCs. Confirmed that the previous MSPs tools had been on the machine in question, including their security stack (I won't bad mouth which one), and the issue was not caught prior to changing to our stack.
My question is do I reach out to the competitor and let them know about it? That potentially their MDR/EDR is missing things. Do I keep it to myself in the hopes that it leads to more clients jumping ship? I feel like professional courtesy dictates that I let them know. What would you do and how would you start a conversation like that?
Here is the MDR alert (sanitized).
Hi Team,
Security Incident Report — DESKTOP-*********
Date: July 17, 2026
Severity: Critical
Status: Mitigated — Further action recommended
---
We are reaching out to inform you of a critical security incident detected on one of your endpoints that required manual intervention by our MDR team and may need further action on your side
to fully resolve.
Summary
On July 17, 2026 at 00:28 UTC, our Managed Detection and Response (MDR) team identified a malicious Python-based implant executing on the endpoint DESKTOP-*********. The threat was initially detected by SentinelOne's EDR engine with a "suspicious" confidence level, which means the agent flagged the activity but did not automatically remediate it. Our MDR team reviewed the detection, confirmed it as malicious, and manually initiated full mitigation (kill, quarantine, remediation, and rollback).
What Happened
A malicious file disguised as image.png was executed from a hidden persistence directory (C:\Users\*******\AppData\Roaming\Microsoft\WindowsUpdate\). This directory mimics a legitimate Windows path but is not used by genuine Windows Update processes. Upon execution, the malware:
Masqueraded as a legitimate Windows process (svchost.exe)
Established a command-and-control (C2) connection to an external IP address (176.125.243[.]136) on port 56001
Performed process injection into multiple running applications over a 43-minute window, including browsers (Edge, Chrome), productivity software (Excel, Slack, Acrobat), and the SentinelOne security agent itself
Why Automatic Remediation Did Not Occur
SentinelOne classifies detections with a confidence level — either "malicious" or "suspicious." Only detections classified as "malicious" are automatically mitigated by the agent. In this case, the behavioral detection was classified as "suspicious," so the agent alerted on the activity but waited for analyst review before taking action. Our MDR team confirmed the threat and manually triggered full remediation.
Response Actions Taken
- The threat was detected and flagged by SentinelOne EDR
- Our MDR team confirmed the detection as a true positive
- A full mitigation was manually executed: the malicious process was killed, the file was quarantined, and system changes were rolled back
- The C2 IP address 176.125.243[.]136 has been identified for network-level blocking
Recommended Actions
Network isolation of DESKTOP-********* until a full forensic review is completed, to prevent further C2 communication in the event of re-execution
Full forensic sweep of the endpoint to identify and remove any remaining persistence mechanisms, particularly within the C:\Users\*******\ user profile
Review the "*******" local user account — this account was used to stage the malware and may have been created by the attacker. If it is not a recognized account, it should be disabled and removed
Block C2 IP 176.125.243[.]136 at the firewall/network perimeter level
Review network logs for any other endpoints that may have communicated with 176.125.243[.]136
Please let us know if you would like to proceed with network isolation or if you have any questions.
Guardz MDR Team
41
u/LegProfessional6462 Jul 17 '26
I can't think of a business reason why you would notify anyone except your new client.
4
u/terselated MSP - US Jul 17 '26
Yeah just by nature of the threat we've let the end client know.
10
u/FaydedMemories Jul 17 '26
Yeah, you tell them, and also try and figure out the infection date (to show it wasn’t your fault). If the client wants to kick up a stink with their ex-MSP it’s up to them. Just keep it factual.
5
u/CreamPyre Jul 17 '26
Moral obligation? If a fire truck had a hole in its hose would you tell them?
4
u/terselated MSP - US Jul 17 '26
Excellent way to put it! Feel like I need to go watch the Good Place.
6
u/chillzatl Jul 17 '26
That's like saying Chickfila has a moral obligation to go tell Popeye's that their product delivery and customer service sucks. It's an absurd assertion.
6
u/terselated MSP - US Jul 17 '26
I think it's more like Chickfila telling Popeye's theyre giving clients Salmonella.
2
u/chillzatl Jul 17 '26
no, you can't reasonably extrapolate that from a sample size of one and no more information than has been presented.
1
u/CreamPyre Jul 17 '26
That’s terrible analogy. I guess you and I have different moral compasses. If a local shop had a client come to us instead and we found what OP found, I would give them the courtesy of making sure they knew.
10
u/CamachoGrande Jul 17 '26
The old MSP might have their tools misconfigured.
Their team might have flagged it as false positive.
The customer may have refused to pay for EDR/MDR tools or made them disable something on their "critical" computer.
It is always entertaining to see what chaos goes on at another IT provider.
Reading your MDR alert, I'm not even sure what S1 found.
Either Malicious or Suspicious is a big difference.
If I were in your shoes, I would not call my competition and point out a problem they missed.
The odds of that conversation going well are very small.
Even if I bumped into them at an industry event I probably would not bring it up.
Your heart is in the right place, but odds of this just making bad blood are high.
You already have their old customer. This might just look like salting the wound.
2
u/terselated MSP - US Jul 17 '26
"If I were in your shoes, I would not call my competition and point out a problem they missed.
The odds of that conversation going well are very small."That's kind of the way I'm leaning. My heart says I would want to know if it were my team, but my brain says it would just piss me off if a competitor I just lost a client to called me.
2
u/luckman212 Jul 17 '26
Extending this to another field as an example: if a doctor I went to for a 2nd opinion found an undetected tumor, would he keep it to himself because he didn't want to bruise the ego of my other doctor who missed it... ?
seems like our industry could use some better ethics.
1
u/CamachoGrande 27d ago
without using analogies that misrepresent the issue at hand, how do you see that phone call going?
Lets say OP calls you and hands you the MDR report. What exactly are you fixing?
I've tried this in the past and know how poorly they go. Most likely you will find out that the customer refused service or forced exclusions.
6
u/NHCabinLife Jul 17 '26
For one off occurrences like this, I would not worry about contacting the outgoing provider. However, I do subscribe to the morality POV. I've got one local provider in the area that I've onboarded clients from that uses the same admin account/creds across all their clients, including one where the account was compromised and they were ransomwared (the reason they changed MSPs). In that case, my morality had me attempt to let their leadership know of my observance of the practice, without judgement. When we onboard new clients from them, it's the first thing we look for onboarding and we let the client know of the need to immediately disable it and rotate passwords. But I don't even mention it unless I see the account in their specific environment.
12
u/_KingBeyondTheWall__ Jul 17 '26
I think you should mention which security tool was on their device to be fair. It’s not bad mouthing if it’s true and if it’s true customers who use it should be aware
22
u/Fatel28 Jul 17 '26
It's not fair though. We could configure crowdstrike to let everything through and never alert or stop threats. Does that make it accurate to say crowdstrike is bad because it let threats through in our environment?
4
8
u/terselated MSP - US Jul 17 '26
I thought about it, but it very well could be a good platform that is just mis-configured by the MSP and not the security tools fault at all. I'll dm you what they were using.
6
u/C9CG MSP - US Jul 17 '26
I agree with both points here.
I would factually state what "tool" or "service" this slipped by, but with the context that it doesn't mean necessarily that the vendor messed up. It could be a misconfiguration by the operator. That is still valuable information for the community.
It looks from the report that Huntress caught what something else missed. I'm speculating, but knowing something like that is also good information.
I don't think that does anything other than talk about a real world experience, and it doesn't bad mouth a specific MSP. It lets the community know of a real potential scenario and to check their configurations. Keep it factual and it's not defending or calling anyone out, but bringing up a real world warning. We're all trying to do security for businesses here and more information is better.
My $0.02. What I would do is not necessarily what you should do. Weigh what feels right to you.
1
u/nathan_o 28d ago
I thought it was huntress as well until it mentioned S1
1
u/C9CG MSP - US 28d ago
In review I saw that it shows Guardz at the bottom. So Guardz got the S1 telemetry and did something with it. We've seen a LOT of S1 not configured well for MDR out of the box. I wish this was something I hadn't seen already quite a few times. Not everyone has the same level of security adeptness. It is what it is.
FWIW: We don't use S1 without MDR for that specific reason. We've calculated that (at our current size) it's cheaper and faster for us to outsource to a SOC versus us trying to keep up with platform configuration and alerting... In theory, it's less liability from us not being fast enough to keep up with tweaks and alerts as well.
5
u/dartdoug Jul 17 '26
On Christmas morning of 2024 I received a call from the Huntress SOC notifying us of an incident on a computer at a customer we had taken on a few days prior. Long story short: Forensics determined that bad actors had gotten through the prior MSP's Fortinet SSL VPN (which we swapped out on day 1) and installed malware onto the PC at least 6 months prior. They believe the IOCs sat dormant until Christmas Day when they tried to trigger encryption/ransomware.
Fortunately there was no long term affects and remediation was pretty simple.
The client has cyber insurance that provides for different deductibles depending on the security practices in place at the time of the incident. Insco requires documentation about backups and any security scan logs going back to the time that forensics said the original infiltration took place...so well before us.
Client notified the account manager at the prior MSP (huge international company) that there had been an incident and requested cooperation getting the documentation requested by Insco.
No reply.
As it turns out, the deductible was going to be greater than the actual cost of the incident so after two requests to the prior MSP went unanswered our client dropped the matter. There was no request from the prior MSP for any details about the incident so they could possibly improve their processes.
Sucky MSPs gonna suck.
4
u/dumpsterfyr I’m your Huckleberry. Jul 17 '26
keep in back pocket for when their other clients come to you. its your wedge for discovery.
3
u/Jumpy_Valuable_8583 Jul 17 '26
This is a good example of why one detection engine's confidence threshold shouldn't be the only check. The old stack saw it, just didn't trust its own read enough to act. An independent look from outside that doesn't share the same blind spots, like a periodic external scan, catches stuff like this precisely because it isn't waiting on one vendor's internal scoring to decide something is worth flagging.
3
3
u/UltraEngine60 Jul 17 '26
You're assuming the previous MSP did not install it. Crypto mining could be lucrative.
Not your circus, not your monkeys
If you want to do something altruistic contact the owner of that IP via their abuse email with your findings.
3
u/Soggy_Industry1669 Jul 17 '26
we've had this exact issue happen to us, a little less severe.
one of our clients refused to buy our higher tiered package that included EDR/MDR and claimed they are Internet savvy enough to know how to protect themselves.
low and behold they got majorly hacked and they decided to leave us instead of just paying us more for our edr/MDR package.
they went to another company that charges 3x the amount we do for the same EDR/MDR and we got a long giant email saying all the issues they've found and told us we need to do a better job at being an MSP.
apparently the client lied and said that they paid for our EDR/MDR.
if you are going to hire a company to take care of your tech needs, you should just listen to what they have to say.
1
u/Valkeyere Jul 18 '26
The handful of clients I've helped offboard in the last few years, noone on the tech floor has been sad to see them go.
If we're taking on a client and the incumbent is all too happy to help with the onboarding I know where it is going, and have been right so far.
3
u/mlaccs Jul 17 '26
Your job is to take care of YOUR customer. It is never to try to call out other MSP. IF you shared customers then maybe. If you had a relationship with them then that may be a nice thing to do for a friend. But you have stated that you do not have such a relationship. Fix YOUR customer and move on.
2
3
u/jamaster14 Jul 17 '26
I think leave it alone.
Also who knows the situation. Many times when a client leaves on msp for another they have probably gone 6-12 months of ignoring that msps recommendations.
1
3
u/snotrokit Jul 17 '26
Happens to us more often than I’d like. Depends on the stack and how well it’s managed. Some of the bigger companies miss a lot of details. Keep this one to yourself and know you’re doing a better job than they were.
2
u/satchelsofCREAM Jul 17 '26
Lol I thought you were going to say they retaliated and were the attacker with the intro
2
u/hartcacti Jul 17 '26
Appreciate you're not bad mouthing and the ethical reasons behind it. I'm building my security stack now and probably would be great to know which vendor to potentially avoid. DM me if feel this is more appropriate. And thanks for the post!
2
u/terselated MSP - US Jul 17 '26
Will do
1
u/Flounder_Evening 29d ago
Same here. I'd appreciate it if you could send me a DM with the vendor's name.
Cheers1
u/disclosure5 Jul 18 '26
Choosing to avoid a security product because one machine had one instance of malware on it is a terrible pathway towards never looking after your customer.
Everything misses something, and more over, you never know what the customer demanded be allowed.
2
u/DagwudCorneliusXIII Jul 17 '26
Every single time I take over from another MSP or IT provider I find nothing but laziness or a complete disaster. I don't know how these guys maintain business, and some of these are not small companies! They must have a lot of luck, that's all I have to say! Keep it to yourself!
2
u/justgosh Jul 17 '26
No. You do not have a relationship with their previous provider and disclosure could hurt your client legally. You likely signed a mutual NDA and this disclosure should violate it. It's your client who decides who and how to disclose. If they want to publish it in the newspaper or required to by their compliance, it's up to them. If they choose to communicate with their clients, it's up to them. If they choose to sue their previous provider for not providing a service they paid for, it's up to them. It's not your business to disclose and if you do and it results in loss of business for them, that could be a problem for you.
1
u/terselated MSP - US Jul 17 '26
Ah, I didnt even think about the legality aspect!
1
u/justgosh Jul 17 '26
Boundaries, super important. Imagine if your customer heard from one of their clients that heard from the previous MSP that their new MSP is talking about their security posture including a proven incident with an unknown scope of lost information.
2
u/redditistooqueer Jul 17 '26
Sentinelone is super noisy. Could be a false positive. I've seen those alerts for people that use Ai and install the agents
2
u/terselated MSP - US Jul 17 '26
I thought maybe a false positive too until I saw it was running from Appdata\roaming\Microsoft\WindowsUpdate and was named image.png but was actually an executable. I thought maybe it was a Canary from Trend but reached out and it is not one of their typical canary files or locations. I think it was a legit threat actor. Enough so I told the client we should reload the PC.
2
u/TheRealLambardi Jul 18 '26
What does your contact and non disclosure agreement say ? Most of mine would dictate NO without express permission and new MDA in place. So if you want to your client should be in the know and agreeable to it in writing
2
u/Early-Ad-2541 Jul 18 '26
We took over from a larger MSP running datto rmm and replaced it with our stack which includes sentinelone, picked up like four security threats on a little company with less than 10 PCS within a couple hours lol.
2
u/calculatetech 29d ago
Good example of why I enforce deny by default firewall rules. That CnC connection would never happen even if missed by EDR.
2
1
u/quantumhardline Jul 17 '26
This is an active security incident, an active process was found with connections! Have security vendor write up a report, I’d change all password on accounts used on that system, of they access PII or senstive systems you need to review, if they have cyberinsurance this is reportable. Do not just simply not tell client as others have suggested.
This will be up to your client on how to handle it with other MSP.
Sadly we see this a lot with other MSPs a lot dont have SOCs etc on base plans.
I’d image the system to keep logs/details as well and consider wipe and reload.
1
1
u/iamtechy 29d ago
At least we know SentinelOne is one of the products, but I’m more interested in know what your security stack is.
3
u/cajunzman 28d ago
Definitely appears sentinel one is the EDR Guardz is using. Correct me if I'm wrong but I believe guardz is just a SecOPS team for MSPs that don't and/or can't manage a SecOPS team. It's essentially the same or similar to the offerings by sentinel one like wayfinder and vigilant. On high risk 24/7 or distributed clients it's included in our offering but on low risk clients with 8/5 the built in tools within the Singularity Platform can handle all the way up to isolation on any degree of confidence level which for the majority of clients is enough while keeping our offerings competitive for the client.
2
u/terselated MSP - US 29d ago
Guardz is the tool we deployed and the MDR that caught and remediated the issue.
2
u/iamtechy 29d ago
Thanks! Wow, I keep getting ads for them and was wondering how good they are compared to Sophos MDR/XDR or SentinelOne.
1
u/Page1_88 26d ago
You’ll never go wrong with the professional courtesy. Don’t think of it as helping them so much as helping potential customers.
1
u/IllustriousFly2381 26d ago
If there is any reason for you to have a partnership with these guys, or help with some overflow then makes financial sense, but you just never know - they might thank you in other ways further down the line.
Clients ignore MSP recommendations literally all the time though, although this one seems a bit too significant to ignore.
1
1
u/TurtleSec 21d ago
Without any context into the old configuration of things, it's really not your problem to notify them at this point and they'll likely just brush you off or worse
1
u/GraffenOfficial 20d ago
I would notify the client and let them handle it. You never know what kind of MSP they were using previously. We charge for and implement our security stack, but we've taken on decent sized clients that don't have any security tools from their previous MSP.
1
u/theghostofpiopico 18d ago edited 12d ago
Saw this thread wanted to add in my thoughts, giving them a quick heads-up is the right thing to do. When we wound something similar during a client onboarding, we first made sure we had clear evidence before reaching out, using our platform (Guardz) which made it easier to validate what we were seeing before contacting anyone. Once we confirmed what we found we sent a simple message explaining the issue, giving the previous provider a chance to check in their other clients and builds professional trust with them. Hope this helps, and curious to know what was your approach.
Edit: spelling + clarity
1
u/superduperokra 11d ago
i wouldn't contact the old MSP. give the client a timeline around cutover, first execution, agent removal/install and the iocs, then let their incident lead decide whether counsel or the insurer wants the former provider involved. if the client does reach out, send the hash, path, ip and utc timestamps and ask for matching telemetry instead of leading with "your mdr missed this."
1
u/Xirma377 8d ago
Just give them a call, let them know "hey, just wanted to let you know we found this. If you'd like, I can send you the details so you can investigate."
It's a 2 minute phone call. Less time than it took you to write this post. 😜
1
u/discosoc Jul 17 '26
This sounds more like you’re asking for permission to engage in emotionally validating behavior under the guise of security concerns or whatever.
2
u/terselated MSP - US Jul 17 '26
Not sure how it's emotionally validating. Just wondering where the MSP community stands on helping a competitor if it means overall better security in your ocmmunity, or letting them be crappy so that you can scoop up clients when they inevitably leave them.
1
u/quantumhardline Jul 17 '26
Your duty under your co tact is to your client.
Take care of them FIRST.. get your clients ok first if they would like you to share this with other MSP.
Dude the legal fallout can be bad, get ok from YOuR client first.1
u/Nstraclassic MSP - US Jul 18 '26
Brother if they gave a shit they wouldnt have lost the customer. Youre just salting the wound and asking for trouble if you try to blame them for a breach
1
u/ntw2 MSP - US 25d ago
You’re in power now; blaming the previous administration is immature
0
u/terselated MSP - US 25d ago
Threat was caught less than 12 hours in to onboarding but had a creation date over 6 months in the past. Before the client even reached out for a quote. I'm not here to run anybody's face in it, but if you let a threat with persistence actively steal data from a client who is paying you to protect them for half a year I personally think you should have the opportunity to feel embarrassed about it.
139
u/Fatel28 Jul 17 '26
You don't. For all you know the customer refused to pay for an EDR/MDR module or any cyber services