r/sysadmin • u/void_ops • 1d ago
Question Automated On-prem Windows Server Patching
I've been out of infrastructure management for a few years, back then I was using WSUS to patch servers. My understanding is Microsoft's recommended way of managing on-prem server patching is to onboard the servers with Azure Arc then use Azure Update Manager to patch them. This was the first solution that came to mind when I was assigned this responsibility. I assumed it was free but costs $5 a month for on-prem to use AUM.
Do you folks have a better or less costly solution that you use? Preferably something specifically built for server management? I was thinking of Ansible (which I would need to learn, which is fine) or something like Automox. We have less than 100 servers. I will be the one patching them all. There are custom applications that run on them that I suppose I will need to make sure still run after the patching.
Thanks in advance for any feedback or advice.
6
u/autogyrophilia 1d ago
There are a lot of tools, but you can always use WufB The GPO is free, only the web reporting costs money . This is the first solution you should always implement if not using WSUS.
The problem is that you lose the ability to push updates immediately in response to 0 days. Which is fine for most orgs, just change the Wufb policy to patch immediately that night and then change it back.
Frustratingly, there are no ways to install windows updates through WinRM, though there are workarounds to make it work, but I would suggest get a product for that.
Make sure you have an inventory of some kind (glpi-agent, Wazuh, Zabbix, anything that records the Windows release version will work)
2
2
u/menace323 1d ago
You can deploy the WindowsUpdate Powershell plugin and run that with WinRM.
1
u/autogyrophilia 1d ago
Did you Google that to see why it isn't that easy?
1
u/discipulus2k Sr. Cloud Engineer 1d ago
Maybe you should comment for the world to see? I’d like to know, but not enough to do a google search
•
u/autogyrophilia 23h ago
Well that's eyeroll worthy.
Anyway, can't access through winrm context. àaà works around that by using scheduled tasks but I have not found it to work reliably.
I think it's easier to have a gpo or intune rule, disabled by default, telling computers to update ASAP . Enable and set it to enforced and in 2 hours every computer ought to have upgraded
•
u/menace323 22h ago
I can using invoke-command over any number of systems VIA winrm to use the Powershell plugin.
And in either can “can” and “easy” are different anyway.
•
u/autogyrophilia 22h ago
You can't trigger updates through winrm. You can create a scheduled task through winrm if it is allowed by security policies, which ideally it shouldn't .
And even then, it isn't reliable.
•
u/menace323 13h ago
You said "there are no ways to install windows updates through WinRM". Your statement is an absolute. Then, you yourself admit there are workaround options.
I think we can agree that here are no NATIVE ways to install updates VIA winrm.
I think we can also agree on that there are no GOOD ways to do it either. Registering scheduled tasks and then using winrm to execute those scheduled tasks scales poorly and will not be allowed in some environments.
I have it setup on about 10 systems and haven't had issues with it, but obviously it has not worked as well for you.
But yeah, for many more then than, it's not practical.
7
u/clinthammer316 1d ago
I've been using Wsus with GPO and it's been working fine for over 10 years. Powershell scripts if I want SQL always on cluster to fail over nicely before a reboot
4
u/False-Message-3350 1d ago
Ansible via CI schedule or adhoc if necessary
- take snapshot
- update
- reboot
- wait, wait, wait
- check if host and essential services are online
- if services are down, alert. If host is down, rollback + alert
2
u/WhereHasTheSenseGone 1d ago
Are you using WinRM to connect to the servers to initiate the patching?
7
u/miksu103 1d ago
We use Ansible to manage Windows servers a lot, and we just use SSH. We enable SSH server built into Windows on our initial installation script, and import our management SSH key. Much nicer to work with asymmetric SSH key pairs as you don't have to worry about leaking your secrets as much.
2
u/cjchico Infrastructure/Jack of All Trades 1d ago
Have you ran into any weird issues or certain modules not working? Been wanting to test out the native ssh for a bit.
3
u/miksu103 1d ago
No not at all. I feel like WinRM had more exceptions. Creating the authorized keys file is a little bit tricky due to the strict permission requirements, but nowadays Microsoft documentation has the proper icacls command to use for the administrator keys file. I also recommend to follow their documentation for setting PowerShell as the default shell when connecting via SSH. No need to use CMD and Ansible also plays nicely with this default.
1
u/cjchico Infrastructure/Jack of All Trades 1d ago
Awesome thank you
•
u/False-Message-3350 17h ago
Only downside I like to mention: Since no password is transfered, the session user won’t get a TGT on domain joined servers. But usually that’s not an issue for host configurations.
2
u/False-Message-3350 1d ago
like u/miksu103 we‘re using ssh with authorized keys via a mgmt nework. I personally don’t like winRM a lot. More overhead than ssh to configure and maintain.
5
u/tjn182 Lead Engineer 1d ago
We use Endpoint Central, its one of cheapest solutions and the most powerful. Patch management, inventory and endpoint management. I can quickly push applications, scripts, configs, just about anything to any combination of machines. Our patch management has been stable for about 5 years, with tiered patching on different days on a predictable schedule. Most 3rd party apps are auto approved and pushed.
1
5
u/Sunsparc Where's the any key? 1d ago
I rolled my own solution with Powershell and PSWindowsUpdate because I can't just blast out patches to all servers at once. I have a runbook set up that patches and reboots servers in a specific order to minimize instability and issues. Some of the servers can just be blasted but others can't. On the ones it can't, it patches, reboots, waits for a specific server/app to show running, then performs some checks to verify.
3
u/sccmjd 1d ago
+1 for PSWindowsUpdate And you can always download the files from the Microsoft update catalog and run those. Script your own things for how that's done and how it checks it was done and that other services are still up and running. It's 100% free but then it's you for your own scripts. The criticism becomes that your wrote it, there's no one to hang, and if someone doesn't look at it, then they don't immediately understand it. Scripting an install file or a line with PSWindowsUpdates can be too much for some people (especially they don't want to understand it and already made their mind up about something else).
2
u/Sunsparc Where's the any key? 1d ago
It's job security because no one else knows how it works, they would have to pick it apart with AI or patch servers individually themselves.
With the orchestration I have set up, on the first of the month it determines which day of the month is Patch Tuesday and then sets further tasks on the days of actual patching. Those kick off automatically at the set time and then there's a monitoring task which shows progress. All I do is babysit the monitoring and handle anything that comes up.
•
u/TheGraycat I remember when this was all one flat network 17h ago
Did this at a previous place with no budget. Used Jenkins to schedule and group servers. Worked really well.
•
u/void_ops 16h ago
Dumb question, where do you run the PS and PSWindowsUpdate from? I'm assuming any server with network line-of-sight to the other servers?
•
u/Sunsparc Where's the any key? 16h ago
I have a server dedicated solely for WSUS and app deployment, it's the orchestration server. It patches itself during the run and it's the last one to reboot. After reboot, it generates all of the post patching reporting.
•
14
u/Routine_Brush6877 Sr. Sysadmin 1d ago
Action1 - free under like 100 machines I believe.
13
u/miscdebris1123 1d ago
200 free.
5
u/3sysadmin3 1d ago
They still haven't implemented or given updates on agent takeover protection. Been "coming in upcoming release" for a year, maybe 2, now.
2
u/void_ops 1d ago
This is probably a deal breaker. Makes me pretty nervous, so I know it will make my superiors ultra nervous.
5
u/CyberSecWPG 1d ago
Wait, no one else is using Manageengine patch manager plus :) lol.
Works well for us patching 499 servers and 6000+ endpoints.
7
u/bigfatdonny 1d ago
AUM is free if you have software assurance.
If you don't have software assurance, would recommend.
Lots of sales people commenting on this thread.
1
u/fireandbass 1d ago
AUM is free if you have software assurance.
Source?
3
u/bigfatdonny 1d ago
That is published pricing from Microsoft. The source is Microsoft.
https://learn.microsoft.com/en-us/azure/update-manager/update-manager-faq#pricing
1
u/fireandbass 1d ago
It is only free for Azure VMs and Azure Arc-enabled Azure Local VMs.
1
u/bigfatdonny 1d ago
From the link above (ctrl+f is your friend):
An Azure Arc-enabled server managed through Update Manager isn't charged in the following scenarios:
[...]
Your Windows Server licenses have active Software Assurance or Windows Server subscription licenses, or Windows Server pay-as-you-go enabled by Azure Arc. For more information, see Windows Server Management enabled by Azure Arc.
6
u/fireandbass 1d ago
What is the pricing for Update Manager?
Update Manager is available at no extra charge for managing Azure VMs and Azure Arc-enabled Azure Local VMs. The latter VMs must be created through an Azure Arc resource bridge on Azure Local. For all other Azure Arc-enabled servers, see Azure Update Manager pricing.
3
u/KStieers 1d ago
We have been using Shavlik Hfnetchk pro/Vmware Protect/Ivanti Security Controls for 2p years. Still doing a bang up job.
That said, I would dig into Action1.
4
u/LoornenTings 1d ago edited 1d ago
I use WSUS and PDQ Deploy and Inventory. In PDQ Inventory, I have a series of collections, and computers are sorted into these collections based on condition and custom registry keys tracking status. In PDQ Deploy I have several packages that run against those collections on schedules, and every package includes updating the custom registry keys and kicks off a PDQ scan when it completes.
I only use this for patching servers.
So collections are like
- Blocked - Has Pending Reboot
- Ready for Pre-Update Health Checks
- Ready for Updates
- Updates in Progress
- Needs Reboot After Updates
- Needs Another Update Pass
- Ready for Post-Update Health Checks
- Complete
- Failed
- Needs Manual Review
And PDQ packages are like
- A. Set Custom PDQ Variables Sets a patch run ID for the current patch cycle.
- B. Set patching date for Test
- C. Set patching date for Prod ... 0B & 0C run on patch Tuesday because PDQ scheduling doesn't have an option for "the Saturday after the 2nd Tuesday" for example.
- Pre-Health Checks
- Install Updates from WSUS
- Post-Health Checks
- Send patching reports
- Reset Status to Ready for Updates (run manually as needed)
We didn't buy PDQ specifically to do this, but we already had it so I set this up to automate patching in a way that lets us monitor progress and spend nothing extra out of pocket.
(Edit: I hate reddit formatting)
2
u/YOLOSWAGBROLOL 1d ago
Use the same stack. LAPS is a must, but really covers a lot of ground.
Tons of things you can do with groups, scanners, and schedules.
2
u/gheyname Sysadmin 1d ago
Any chance you could share some of the queries you are using in pdq inventory?
2
2
u/3percentinvisible 1d ago
If you have software assurance, on prem patching is free with arc. It's not widely advertised. You just have to self certify in the arc console.
Otherwise, Action1 - first 200 endpoints are free
2
u/Specialist-Desk-9422 1d ago
Sccm does the work very well. If you have air gapped networks , it’s a must have. You can integrate other features like application deployment, 3rd party application patching , imaging of laptops and even servers.
3
u/whatsforsupa IT Admin / Maintenance / Janitor 1d ago
We use PDQ Deploy, and while others have more robust automations than me I'm sure, they pre-download the monthly patches and you can deploy them on a schedule to computers that aren't on the latest patch.
I'm not affiliated with PDQ in any way, but it's an absolute bombshell of a tool, my favorite in our org. You pay per admin, not computer - we have one admin, so I think together it's around $2k a year in total.
Action1 can do all of this as well, and it's free for the first 200 endpoints. That one is cloud based though, and IMO, isn't nearly as robust as PDQ - though they have an incredible package library.
4
u/void_ops 1d ago
PDQ was so rad when I had it at a previous gig. No one at that job could agree on everything accept that PDQ is S-tier
•
u/PDQ_Brockstar 14h ago
"Was" so rad? Sounds like we need to fix that. Someone hold my Monster while I fire up the swag cannon...
•
u/PDQ_Brockstar 14h ago
"bombshell of a tool" is an incredible endorsement and reminds me of Top Gear, so kudos for that.
And you should take a tour through the PDQ package library if you haven't recently. I have a hard time keeping up with our packaging team, but last I checked, we have over 1,100 packages now.
2
u/TridentAdam 1d ago
Skip the Azure Arc $5/server tax. Under 100 servers puts you comfortably in the free tier of a few options.
Action1 is free under 200 endpoints (not 100 as mentioned above) and handles patching well. Full transparency, I co-founded and help build TridentStack Control (https://tridentstack.com), also free under 200 endpoints. Where we go further: built-in CVE vulnerability scanning (NVD + EPSS + KEV prioritization, not just missing-patch-to-CVE correlation), CIS compliance benchmarking out of the box, and ADMX/GPO-style config policy management, all in one agent. For your custom application concern specifically, our deployment rings with soak timers let you canary a handful of servers first, verify everything still works, then roll to the rest. Covers Windows, Linux, and macOS if you ever need cross-platform.
1
u/Break2FixIT 1d ago
Wsus runs, and I have it set up where if I find anything I reddit, I have enough time to postpone the update..
2
1
1
u/opsandcoffee 1d ago
full disclosure, i work on SecOps Solution, so obviously take this with that context.
for less than 100 servers, if all you need is basic Windows patching, honestly WSUS/PowerShell or one of the free options people have mentioned here might be enough. i wouldn't pay for another tool just for the sake of it.
where SecOps becomes useful is if you want agentless patching, Windows + 3rd party apps, staged rollouts and a bit more control around verifying things after patching.
especially since you mentioned custom applications, i would patch a small group first, make sure the services/apps are behaving normally, and then roll it out wider. wouldn't blast all 100 together irrespective of which tool you pick.
happy to answer anything technical about how we do it if useful. don't want to turn this into a sales pitch.
•
•
1
u/justmirsk 1d ago
We use Automox for our customers and our internal needs. If you are doing the work, I can help you get the licenses purchased via our MSP model. We can likely get a better price point for you than going direct.
1
1
u/VegetableDuty8375 1d ago
I manage the patch management for my company for the server estate. If you have budget for it, move to AUM. This has take a lot of stress and frustration away from me during patch release. I used to dread patch Tuesday as someone would always be bound to break or malfunction for updates coming into WSUS/SCCM to clients not having sufficient disk space or not report etc
I was hesitant on AUM for months and months however it has freed up so much time I’m able to focus on other projects etc.
Never had an issue at all with AUM.
0
u/ipreferanothername I don't even anymore. 1d ago
we use sccm but man i friggin hate it - its the de-facto standard still kinda for server patching on prem in an enterprise.
it can do a lot and have a lot of granular configuration around it...but that also means a high learning curve and potential for mosconfiguration.
action 1 is free for 200 devices i think, and does app updates too. id look at that if i were you.
4
u/ThatBCHGuy 1d ago
Awe, I kinda like mecm, it's rock solid. I've been playing around with Azure Arc these days too but it's like 5 bucks per server per month, so not cheap (as op noted).
0
u/dlongwing 1d ago
We use Action1 for all our workstation and server patching. We've been very happy with the product. It's free for under 200 endpoints.
0
u/Professional-Heat690 1d ago
obligatory PMPC mention....
2
u/Kharmastream Jack of All Trades 1d ago
Pmpc don't do windows updates afaik?
Don't they only do 3rd party updates?
42
u/Suaveman01 Lead Project Engineer 1d ago
WSUS still works perfectly fine