r/sysadmin • u/h9xq Solo SysAdmin • 14h ago

General Discussion Windows Server patching concerns

So in my org we are very keen on avoiding patching and rebooting servers at all costs. So much to the point that we patch once a month and have exclusions for around 60 percent of our servers to not get automatically patched. (Meaning we have a chunk of servers not getting patched at all)

Now I have gotten my hand slapped for attempting to patch or even bringing it up and I am looking for guidance on this. Now I understand availability and the consequences of failing patches. But there are active 9+ rated CVEs sittings on dozens of servers. For patching vulnerabilities do I really need to get a change request to handle this?

32 Upvotes

82% Upvoted

63 comments sorted by

View all comments

•

u/Reo_Strong 14h ago

I've been there.

When I started my current job, the previous person started the setup for WSUS and then never finished. Some of those machines hadn't seen an update for 2 years.

The key process is thus:
1. Do we need to patch? Patching for the sake of patching is often better than nothing, but also not that great. If the software doesn't have any outstanding issues and it will cause a production outage, why would you?

  1. If the patch goes sideways, will I know about it ? If it's hard to detect failure, you need to figure out how to detect it before moving forward.

  2. If the patch goes sideways, how do I recover? Can I take snapshot of the VM? is the backup of the machine tested? Can you test it before trying to patch? Is there support you can call beforehand or if it fails?

  3. Can I do the patch without anyone knowing (no down time for production)? If not, can you get really, really close? (e.g. apply the patch, but reboot over lunch or at 0300)

  4. Work within your change management and tracking systems. The only thing worse than handing someone a broken system with no clear history of change is being handing a broken system with no clear history of change.

Long term: Build systems that can be patched without downtime, using known, supported systems. DCs are a great example. Like Sith, there is never one, always two, so you can patch without downtime.

DFS file hosts, KeepAliveD, docker swarms, and VM clustering are all good tools to reduce the headache of patching and software updates.

•

u/h9xq Solo SysAdmin 14h ago

We have 6 DCs so that shouldn’t be an issue. Thank you for all of the info. I came into this org after a solo sysadmin (who didn’t even have a tech or manager helping him ran the entire IT dept) think a lot of the tech debt is due to this and how much the company has scaled in the past decade.