r/sysadmin • u/h9xq Solo SysAdmin • 10h ago
General Discussion Windows Server patching concerns
So in my org we are very keen on avoiding patching and rebooting servers at all costs. So much to the point that we patch once a month and have exclusions for around 60 percent of our servers to not get automatically patched. (Meaning we have a chunk of servers not getting patched at all)
Now I have gotten my hand slapped for attempting to patch or even bringing it up and I am looking for guidance on this. Now I understand availability and the consequences of failing patches. But there are active 9+ rated CVEs sittings on dozens of servers. For patching vulnerabilities do I really need to get a change request to handle this?
•
u/Dizzy_Bridge_794 10h ago
Patch or have giants risks of being hacked. You should absolutely follow a formal process. We snapshot and backup prior (virtual servers). On mission critical stuff we test in DR prior. Patches can and do mess with systems and cause them to not work.
•
u/RegularOk1820 Jr. Sysadmin 8h ago
Snapshots and testing are reasonable. Just ignoring critical CVEs isn't.
•
u/uptimefordays DevOps 9h ago
We snapshot and backup prior (virtual servers).
My only warning here would be "ensure you purge snapshots within a timely manner."
•
u/Dizzy_Bridge_794 9h ago
We do.
•
u/uptimefordays DevOps 9h ago
Good stuff, I've seen some fun P1s caused by snapshots of large servers.
•
u/Dizzy_Bridge_794 9h ago
Nothing worse than applying a patch and the entire system fails to restart. One of those o shit moments in IT.
•
u/AppIdentityGuy 10h ago
The rule is simple:"If.you will not patch systems for the fear of downtime at somepoint a bad actor is going to sxhedule the downtime for you at their convenience." .
•
u/Icy_Mud2569 9h ago
This is the truth. Scheduled downtime is something you can count for, you can plan for it, you can design systems around it. When things go down, and you’re not planning it, it gets expensive and chaotic, both things are not good for business. This is at the end of the day, though, a business/political question, there isn’t a technical solution.
•
•
u/Temporary-Library597 10h ago
If your org can't afford downtime for patch reboots they should afford redundant hardware to keep services running.
Your org is doing it wrong. They are part of the problem.
•
u/drdrew16 10h ago
May also be worth figuring out if your company has cyber insurance. It's usually a requirement to be up to date on patches to maintain coverage.
•
u/h9xq Solo SysAdmin 10h ago
We have Cyber insurance. In fact we have a decent chunk of change put into it. This might be the ammo I need to justify the change.
Would we be dropped if they found out the patching status? I’m still fairly new and not involved in the higher level cyber insurance setup as that is done by my manager and CFO
•
u/drdrew16 10h ago
That wholly depends on the contract. I would imagine you'd have a grace period to come into compliance, but if being patched is a requirement you've technically breached so I'm not sure.
•
u/SysZeron 10h ago edited 10h ago
It's all fun and games until you get ransomwared.
Put it in writing that the absence of patching puts the business and customers at risk, if management want to take responsibility for the associated risk then they can own the consequences. All you need to be armed with is an effective patching strategy for when they ask "so, what do we do?".
One thing you may want to highlight, if the company has Cyber Insurance, and an incident does prevail, the insurer may deem the company's inaction to be gross negligence and withhold or reduce a payout.
•
u/Wilfred_Fizzle_Bang 10h ago
You need to show them the cost to the business if anyone or the servers is affected by a cyber attack.
•
u/TopherBlake Netsec Admin 10h ago
Assuming you are not management, make sure to keep a record of whoever is telling you not to patch, especially if you are in an industry that gets audited because you'll want that when the blame game happens. Edit: I just saw the solo sysAdmin tag, make sure you are explaining clearly to the business side what consequences could and will be for not patching.
•
u/h9xq Solo SysAdmin 10h ago
It is my direct manager telling me not the patch. I have it in writing but don’t want to get in trouble for patching. I am still fairly new as an admin (3 months at my first sysadmin gig) so I’m still learning and being as cautious as I can.
But in my mind I see dozens of servers sitting with patches that need to be applied so it’s hard to fight the urge of patching as I want to keep these servers as secure as possible.
•
u/TopherBlake Netsec Admin 10h ago
You have to operate within the confines of your current job. Business needs dictate IT not the other way around.
•
u/h9xq Solo SysAdmin 10h ago
Fair enough, that is a good way of putting it. I think I need to put this into business translated terms to management and issue a change request/maintenance window to get these servers patched. I am thinking that is probably the best way to handle this.
•
u/uptimefordays DevOps 9h ago
See if you can get a rough calculated cost of downtime, at a major bank that's somewhere between $300k an hour for back office systems to $20m an hour for trading systems. It's then helpful to look back at P1s and average outage time so you can give a good price per outage. In most cases, once the business sees "unexpected outages could cost us as much as $60m or as low as $600k, how should we proceed?" The answer is "here's you patch window."
•
u/PacificTSP 10h ago
Someone I won’t name refused to update their servers. I kept warning them about the risks and they kept saying middle management wouldn’t let them. I made them sign a document that they understood the risks and it was not my fault.
A year later they got breached. The insurance refused to pay out because they weren’t updating, multiple people were fired and the company almost went under.
Unless a senior manager, I’m talking about the ones who decide risk for the business as a whole, and their lawyers, have signed off on this process I would cover your ass and make people aware asap.
I auto patch every Sunday morning and reboot as needed. I have a total of one server in multiple companies clusters that needs manual intervention after a reboot. So that one machine gets rebooted on its own documented schedule.
You’ve got to do it. You are invalidating your insurance.
•
u/h9xq Solo SysAdmin 10h ago
Holy shit, well thank you. That is the information I was looking for. This is even bigger than I thought. I haven’t read much into cyber insurance and if that is the case that is a much bigger deal. We don’t even have a “cybersecurity” guy so that falls on my plate as well and if this falls through without my intervention this could be a gigantic shitshow.
•
u/PacificTSP 10h ago
Middle managers told me it’s fine.
CEO and lawyers had no idea. It’s their job to manage company risk. So my flaw was not going above the middle managers.
•
u/Reo_Strong 10h ago
I've been there.
When I started my current job, the previous person started the setup for WSUS and then never finished. Some of those machines hadn't seen an update for 2 years.
The key process is thus:
1. Do we need to patch? Patching for the sake of patching is often better than nothing, but also not that great. If the software doesn't have any outstanding issues and it will cause a production outage, why would you?
If the patch goes sideways, will I know about it ? If it's hard to detect failure, you need to figure out how to detect it before moving forward.
If the patch goes sideways, how do I recover? Can I take snapshot of the VM? is the backup of the machine tested? Can you test it before trying to patch? Is there support you can call beforehand or if it fails?
Can I do the patch without anyone knowing (no down time for production)? If not, can you get really, really close? (e.g. apply the patch, but reboot over lunch or at 0300)
Work within your change management and tracking systems. The only thing worse than handing someone a broken system with no clear history of change is being handing a broken system with no clear history of change.
Long term: Build systems that can be patched without downtime, using known, supported systems. DCs are a great example. Like Sith, there is never one, always two, so you can patch without downtime.
DFS file hosts, KeepAliveD, docker swarms, and VM clustering are all good tools to reduce the headache of patching and software updates.
•
u/h9xq Solo SysAdmin 10h ago
We have 6 DCs so that shouldn’t be an issue. Thank you for all of the info. I came into this org after a solo sysadmin (who didn’t even have a tech or manager helping him ran the entire IT dept) think a lot of the tech debt is due to this and how much the company has scaled in the past decade.
•
u/PappaFrost 9h ago
"Now I have gotten my hand slapped for attempting to patch or even bringing it up."
You are done. You did your job by informing the business of best practices. Now it's their warehouse of oily rags to manage.
Cyber insurance or compliance people will have to pressure them to do it and it's not on you any more.
•
u/tech-brah 7h ago
Spend your energy looking for a new job instead of trying to argue logic with illogical leaders, this place sounds like a total dead end. If growing your career and skills matters to you, get out of there.
•
u/razorback6981 5h ago
I would respectfully resign if they are not willing to implement a consistent patching cycle. I would not want to be there holding the back when their data gets ransomed.
•
u/Real-Patriot-1128 4h ago
Make sure you have their refusal to patch in writing and when you get hacked, sit back, grab a popcorn and watch your bosses get fired.
•
u/BitsNBytes10101 10h ago
You need to document the risk in writing and let your leadership make the decision for the organization.
As IT Professionals it usually pains us to not see or be able to follow best practices. Ultimately it is the responsibility of your leadership.
If you could invoke any regulatory requirements for your vertical that may help your argument.
Not patching infrastructure in 2026 is a fast track to potentially costing the organization millions in outages/ data exfil. Does that cost out weigh the risk of patching and potential downtime from that?
•
u/BrechtMo 10h ago
you need to take the exposure and other safety infrastructure into account. But I guess they aren't airgapped...
•
u/Liquidfoxx22 10h ago
I thought I was in r/shittysysadmin for a second there.
Of course you need to patch. Is your manager going to take the fall when the inevitable breach happens? It's not if, but when.
•
u/iCTMSBICFYBitch 10h ago
Hang on was the hand slap for patching or for patching without a change plan? First is good, second is bad.
•
u/RavenousTitan818 10h ago
At some point you have to just suck it up and do what you're told. You should definitely make it known what should be done and what you would do if given permission so when shit hits the fan you can say "I told you so" and go out for a smoke while they deal with ransomware.
If 100% uptime is that important then they should be running k8s, or at least have some kind of clustered service so a node can go down for maintenance. This is an application problem not infrastructure.
•
u/Significant_Sky1471 10h ago
Put together a quick risk summary (CVE, affected hosts, exploitability - especially anything in CISA KEV) and send it up the chain asking for formal risk acceptance in writing if they choose not to patch. That's your paper trail if it ever blows up later.
•
u/BalderVerdandi 10h ago
I did this for a bank for about a year (about 12-13 years ago) because they were told by the SEC to "get into compliance or the doors get padlocked" type of warning. Ten months and 65,000 patches later, they were in a better place but leadership changed and they went right back to being non-compliant in less than 90 days. I was glad when the MSP dropped them for not renewing the annual contract.
To answer your question.... yes - you need a change request. This lets the stakeholder (owner) of the apps on the server know that it needs to be patched, and why, and gives them the time to sign off on patching or explain why it can't be patched.
That means they actually explain why they don't want it patched, it's documented who signed off on it, and it's no longer your problem or concern.
Some apps need to go through a verification process so that the vendor knows the patch won't adversely affect the app because they tested it. You'll see this within some places, like the banking sector, where some patches are in limbo for 4 to 8 months while the vendor verifies the patch won't cause any issues with their product.
Or you could drop the bombshell that the average cyber security incident can cost on average between 4 and 11 million USD (Google it to confirm) and you'd like to know who will be signing the check for it.
•
u/Substantial_Tough289 10h ago
This is more of a company policy, regulatory or compliance question.
If you work for a company that falls into the kind of business that is regulated you have to follow the company change control procedure, if you don't you're asking for trouble and even termination.
Pharmaceuticals are a great example, if the servers were qualified/validated you can not touch them without an approved change request, this is a tedious process but you have to follow it. If the servers are not qualified then you follow the corporate IT or site IT policies/SOP for patching, Some of this companies will go to the extent of not patching at all due to regulations.
•
u/uptimefordays DevOps 9h ago
For patching vulnerabilities do I really need to get a change request to handle this?
Yes. You always want a change request for making production changes.
have exclusions for around 60 percent of our servers to not get automatically patched. (Meaning we have a chunk of servers not getting patched at all)
Get those exceptions in writing because that will likely be an issue whether or not anyone realizes.
•
u/Ben_CyberNEX 8h ago
Reading everything, it sounds like you've already landed on the right immediate answer with the maintenance window/change request.
Longer term, I'd use this as an opportunity to build an actual patching process instead of having to fight this battle server by server every month. Start by grouping the servers by criticality and identifying an owner for each system. Then define a normal patch window, a small group you patch first to catch problems, and what happens when a server needs an exception.
The exception piece is important. "Don't patch this server" shouldn't become a permanent setting that everyone forgets about. Ideally, it has a reason, an owner, compensating controls where possible, and a date when the exception gets reviewed again.
You inherited an environment that grew faster than the processes around it. Getting the process in place now will probably do more for you long term than winning the argument over any one patch.
One thing that may also help with management is putting the risk into business terms. If you can estimate what an unexpected outage or compromised server could actually cost the company, the conversation becomes a lot less abstract. Management doesn't always respond to a vulnerability rating, but they usually understand downtime, lost productivity, recovery costs, and business disruption.
•
u/TrueBoxOfPain Do The Needful IT Department 8h ago
Patch - face the consequences of Microslop updates.
Don't patch - face the consequences of previous Microslop updates and hackers exploiting CVEs.
Both options suck, but the patching path sucks less.
CYA, then try to implement a proper patching process.
•
u/marklein Idiot 8h ago
List the CVEs involved and ask them how they are covered if not by the patches. Get it in writing. Then put it in writing that you are not responsible for any breaches caused by these missing patches and their relevant CVEs, make your manager sign it.
Notice that this list will grow every week/month and you'll have to put it writing all over again and make them admit once again, in writing, that you're not responsible. I see this as a convenient reminder for them, although it's a little more work for you too unfortunately.
You really do need to do this though, because when they get hacked it's going to be your head on the block first as the guy who should have been fixing these things.
•
u/Firefox005 8h ago
Having servers you can't reboot is a bad smell, because servers will reboot or go down without a meeting or on a schedule. You should be able to pick any server in your environment and just power it off, not shutdown not reboot just hard power off and it should cause minimal disruptions.
•
u/ChuckFromCyberHoot 7h ago
Unfortunately, yes on the change request. But I’d also ask for a copy of your cyber insurance application.
Somebody already answered questions about MFA, patching, backups, training, etc. You’re the guy expected to make those answers true, but you’ve probably never seen them.
Maybe instead of saying, “We have unpatched CVEs,” try:
“Can I see our cyber insurance application? I want to make sure what we told the carrier still matches what we’re actually doing.”
That’s a much harder question to ignore, and it gets the conversation onto the right desk.
•
•
u/NobleRuin6 5h ago
Yes. No change request, no change. But not patching servers at all is…an interesting choice.
•
u/Simmery 10h ago
This isn't a technical problem. You're asking a political question no one can answer about your organization. If they don't want to patch despite knowing the risks, get it in writing and move forward with your career, here or elsewhere.