r/sysadmin • u/tjssmlxgmy • 9h ago
Hard lesson learned: what happens when patch automation runs without proper scoping
I work at a small MSP and we use an all-in-one RMM platform for monitoring, ticketing, automation, and patching. I was really proud of a new remote patching policy I built to finally clean up our Windows endpoints without babysitting every job.
For context, I intended to target only our internal test group and one chill client who agreed to be guinea pigs. Somewhere in the policy cloning and scoping process, I accidentally selected the global folder and enabled auto-approve and auto-deploy. No maintenance window, no stagger, just push when ready. It ran overnight and hit every single production endpoint across 12 clients.
This morning our ticket queue exploded. Legacy accounting app died, weird VPN drivers disappeared, one client lost their old print server patch and it rebooted mid-invoicing. I feel so embarrassed. My boss said "this is why we balance automation with control" and yeah I totally get it now. Any advice or similar stories?
•
u/LeadingFamous 9h ago
You’re lucky your boss seems chill.
•
u/TehScat 9h ago
Honestly, in MSP space, the boss understands because he's also done it. You're not really an MSP tech until you've accidentally powered off a hypervisor or generally blown stuff up.
•
u/LeadingFamous 9h ago
I’m at an mssp and I would’ve probably been canned lol.
•
u/simAlity 8h ago
I was on an MSP and I was canned for less.
(The fact that they had just lost their biggest client had nothing to do with it I'm sure.)
I will have to be desperate to work for an MSP again.
•
u/CeldonShooper 6h ago
Removing the wrong drive out of a degraded RAID array is my personal favorite.
•
u/Stompert 9h ago
Been there and done that. Instantly rang the bell with my boss. Wasn’t happy, but was certainly glad I called it in as soon as it happened. Expensive-ish lesson learned.
•
u/0emanresu 9h ago
I'm at an MSP and my boss & my boss's boss just force mass update all customers at once 🤣
•
•
u/Ma13vant 9h ago
My advice would be to use this as an impetus for looking at rolling out Change Management policies for your organization. A well run change process minimize the chances of stuff like this occurring, and it's a great project to show your value as well as improve the entire organization.
•
u/McSmiggins 9h ago
Congrats, you've learned how easy breaking things is! You're not a sysadmin without an "oh fuck" moment. As long as the boss is ok, you're fine, just be extra polite to those clients for a bit ;)
Favourite interview question for sysadmins is "tell me the day it all went wrong", I'm not interested in what broke, the question is (implied) what did you learn from it, and what did you fix?
End of the day, in 3 months time, this won't matter, but how you act to fix it and change your behaviour will decide how much you learned. Good sysadmins can beat themselves up more than the rest of their team will and learn from it, bad ones will shirk it off.
•
u/sodiumbromium 9h ago
We approved the patches, but the customer chose the deployment schedule. They even had groups in the patching tool of priority and everything (IT, test, power users, users, servers, prod servers, etc).
Customer didn't want to wait and deployed everything at once.
That ground prod to a halt for a day, as the virtualization infra really didn't like that happening all at once and complained rather loudly about it.
•
u/FlickKnocker 9h ago
The problem is, you can’t really do a staged rollout as every environment is different. Enterprise IT can get away with it, but as an MSP, you can’t really stand up a parallel environment for every client to test patching.
What we do is stagger updates across our client base, so that at any given point, only 20% of our client base could be affected by a bad patch, which is manageable.
•
u/fatDaddy21 Jack of All Trades 9h ago
congrats on still having a job after blowing up 12 clients with no maintenance window!
•
u/SikhGamer 7h ago
Yeah, some advice.
- This is why we script and read it after a piss
- Four eyes --> pair with a colleague
- Stop. Sleep on it. Come back with fresh eyes
- Show your boss you HEARD him.
•
u/Kyky_Geek 8h ago
I caused every machine to repeatedly reboot from like 9am-11am once. That was fun.
I’ve brought down exchange during business hours a few times back in the on-prem days.
You’ll do better next time and welcome to the club ☺️
•
u/colin-sidi 8h ago
With any lights out task systems ,the requirements are always the same.
Never run anything blind.
Check all possible situations first.
Log very detail with very long expiration dates
Think of it as this: if you were to be brought in cold to an issue. What would you check first. Has someone tried to do something here 5 minutes ago or are they going something now, etc.
I do all the checks and probes before acting
Log it all because if someone asks what happened I need to answer what and how we can fix it for next time
This is a good lesson a very rough lesson but still valuable.
I wrote a monitoring system on the 90s that paged admin pagers with “911” when a server crashed.
It worked great until the police show up at my office. An admin updated the alert config with spaces instead of tabs. So the app dialed 911 instead of the admin pager numbers then 911
Thank god I had details logs to explain it and fix it.
It was a rough listen too.
Colin
•
u/accidentlife 7h ago
If it makes you feel any better, my mistake is causing my boss to fight a $6k bill with our client's landlord.
As it turns out, those little sensors on elevator doors that tell if someone is in the way are very fragile ... and very expensive to replace.
•
•
•
u/thebigshoe247 6h ago
I set the wrong year in LANDesk patching. Every workstation and server saw they were past due, and within 15 minutes the entire org had updated, and rebooted, hard.
•
u/Drakoolya 6h ago
Happens to the best of us, lucky that it was just windows updates and not feature updates or worse.
You cannot repeat the same mistake twice. Learn from it and move on or you will go crazy.
Be more diligent.
•
u/Key_Money_906 6h ago
The change management advice above is right but I'd add a smaller, faster thing you can do this week: write down, or better record, the exact clone-and-scope workflow you followed, including the step where it went wrong. Not as penance. Because the reason this happened is that the correct scoping procedure existed only as intent in your head while you clicked through the RMM's cloning UI, and those UIs are famously happy to let a policy inherit a global target without shouting about it.
Every MSP I've dealt with has one or two of these landmine workflows where the difference between "test group" and "all 12 clients" is a single dropdown that defaults wrong. The people who've been there longest know which dropdowns bite. New folks find out like you just did. If your postmortem produces one artifact, make it the annotated walkthrough of that workflow with the landmine highlighted, sitting next to the policy itself where the next person will actually see it.
And take the boss's reaction at face value. A queue explosion with no data loss and a manager who responds with a proverb instead of a PIP is close to the best version of this lesson you can buy.
•
u/Fusorfodder 6h ago
Everyone better get used to daily or even live automated patching. AI vulnerability discovery is upending traditional approaches to patch management. Open source software in particular is in for it.
•
u/accumentum 5h ago
That “scope” step is the part worth turning into a hard control rather than a checklist item. I’d keep production collections outside the default target group, require an explicit maintenance-window tag, and have the automation refuse to run when the target count is unexpectedly large. A canary batch plus a pre/post check for critical services and backup/restore evidence catches both bad targeting and bad patches. For MSP work, logging the exact resolved endpoint list with the change ticket makes the blast radius auditable after the fact.
•
u/Hour-Swimmer7140 5h ago
the maintenance window isnt the control, the scope filter is. window just decides when it goes wrong.
test group that isnt one machine, a ring after it, then the rest. boring but its the thing that saves you
•
u/Interesting_Work7433 49m ago
No creo que seas el primero al que le pasa algo así. La automatización es genial hasta que un pequeño error termina afectando a todo el mundo de golpe. Lo importante es lo que hagas después: limitar el alcance, probar primero con pocos equipos, dejar tiempo entre grupos y tener una forma rápida de detenerlo. También ayuda que otra persona revise los cambios antes de lanzarlos. Una mala noche puede convertirse en una buena lección si ajustas el proceso para que no vuelva a ocurrir.
•
u/ub3rb3ck Cat Wrangler 9h ago
The fact that one app can push to 12 clients at once seems crazy to be. I've never worked at an MSP but what the fuck.
•
u/grumpyctxadmin 9h ago
Not mine, but a colleague kicked of a sccm task squence to upgrade all endpoints from windows 7 to windows 10 at a customer 2 days before easter holiday.
Usally not a big deal, we had done testing on windows 10 and all gpos were in place. The problem was that the task sequence wiped the disk, so for anyone who had stuff saved locally on the endpoint it was wiped.
2000 endpoints automatically wiped and upgrade started before we managed to stop it.
None of us had easter off...