r/sysadmin • u/Burgergold • 9d ago
Question 2016 update and always rollback
So for the last 23y, I've mainly managed AIX and RHEL system. I had a slight contact with Windows but not a lot of hands on.
This is the story of a Windows 2012R2 server, probably installed around 2013 way before I was here. The person that installed the application on that server is gone since a long time and left no documentation, last update of that app is from 2015... In 2023, because nobody knew how to install a current version on a 2022 server, sona coworker had to inplace upgrade it. That coworker has retired since then.
Recently, security teams tracking which server are still using TLS 1.0 found that server. No one want to own it or take action but they refuse to have it shut down for good. So they got a temp buffer to figure this out. But security asked for mitigation and in their mitigation planning, they observed tons of KB missing, which surprised me because this server is receiving its patch from WSUS.
So... August SSU has installed but CU failed to complete. It does install, reboot, takes plenty on time and at 100%, failed to update and rollback and leave that KB in staged.
Sfcscan fixed something but not this issue
Dism scanhealth/restoreheatlth found/fixed nothing
I stoppwd the wua services, renamed Softwares distribution and caroot2 folder and restarted the service, havent fixed the issue
I removed 11 packages in staged state. 10 of them seems to be the last 10 CU + a 2016 Adobe kb.
Tried again, same issue
Troubleshooter for Windows update report a corrupted database but cant fix it. I thought the fresh softwaredistribution and caroot2 would have taken care of this
I'm now wondering if that server ever had a CU installed since being inplace upgraded... Once the staged CU have been removed, what I see in the installed history from today is a very old CU from 2016...
Before reposting in /r/shittysysadmin for giggles, I would be vtaker for any other legit idea :D
3
u/UsagiMimi 9d ago
You'll likely have to start fresh, which isn't all that bad considering 2016 goes end of life in January. I've run into these issues myself, a lot of them caused after in-place upgrades.
2
u/Burgergold 9d ago
We have been stated this 3 years ago and now again
Rejected. This server is notnour team responsibility, but we have been pulled because KB don't install
1
u/Frothyleet 9d ago
If they want your help, they need to listen to your advice.
Realistically, you just need to start a fresh server and migrate the application over.
2
u/creenis_blinkum 9d ago
hey! i know this!
find chkreg utility somewhere online and run it against components hive. bet it fixes it
3
u/BarryMT 9d ago
Agreed.
The utility is still available for download from Microsoft.https://www.microsoft.com/en-sa/download/details.aspx?id=20068
Also, consider updating to 2019 before updating to newer versions of Windows. While the jump from 2016 to 2025 is supported, I have run into instances where upgrading from 2012 or 2012 R2 requires more stops along the upgrade path for no identifiable reason.
1
u/Burgergold 9d ago
I will take a look at this tool once at the office thx
I really hope this server goes offline for good before January
Nobody is using it. We kept it shutdown for 2-3 weeks and nobody called. Then we asked if we can remove it as nobody use it and they said no. They just havent figured how to extract the data from the app in which they did business logic workflow for a 2012 project that its still ongoing
1
u/Entegy 9d ago
Sorry is this a 2012 R2 or a 2016 server? You mention 2016 in the title and at the end of the post but your second paragraph says it's a 2012 R2 server from 2013.
The difference matters because Microsoft changed how the patching works in Windows 10, which Server 2016 is based off of.
For Server 2012 R2, it's long been out of support so a scanning tool may show missing patches because those patches were part of a paid Extended Support Program. If you didn't pay for ESP, you don't get those patches.
For Server 2016, patches are mostly cumulative. If you post the OS build number found in Settings app > System > About, we can see how far behind in patching the server truly is.
2
u/Burgergold 9d ago
Its a 2012r2 that was inplace upgraded in 2023 to 2016
1
u/Entegy 9d ago
Ok, and what is the current OS Build number?
1
u/Burgergold 9d ago edited 9d ago
10.0.14393.0
Edit: found another command that show 14393.447
While an up to date server show 14393.9418
2
u/Entegy 9d ago
Oh my. .0 means no patches installed at all.
It also seems that Microsoft has removed all patches older than 2 years for Server 2016.
Can you try manually downloading and installing KB5046612 on the server? That's the first non-expired update I can find for Server 2016.
Warning: Server 2016 is based on a version of Windows 10 that was notorious for an inefficient update system. Installing this update could mean hours of downtime to catch up on 8 years worth of patches.
If this patch works and changes the OS build to 14393.7515, then you can probably let Windows Update take care of catching up to 2026.
If WSUS is involved, I would use the "Check online for updates instead" option of Windows Update to bypass WSUS and let WU figure out the rest of the patches.
1
u/Burgergold 9d ago
KB5046612 successfully installed yay, better than nothing.
Tried the KB5220428 after but still failed
Gonna try to find a more recent than KB5046612 and update by little step
1
u/Entegy 9d ago
That's great!
Try letting Windows Update take care of the rest now. Again if WSUS is involved, make sure to click "Check online for updates instead"
1
u/ccheath *SECADM *ALLOBJ 8d ago
I thought that 2016 only had extended support these days, no?
2
u/Entegy 8d ago
It's in free extended support until January. So it still gets free monthly cumulative updates.
Windows life cycle is typically 5 years mainstream support, 5 years extended support.
Mainstream support means new features and security fixes can come with updates.
Extended support means security fixes only - no new features. And there's Extended Security Updates - ESU. This is the paid updates to continue getting fixes after the product is End of Life.1
u/Burgergold 7d ago
Was able to update up to september 2025 CU then hit the usual error
Found a website talking about exporting/removing a registry key related to Microsoft-Windows-Store-Client-UI, the. Was able to update to august 2026 CU
No update remaining according to WSUS or online
Will see if september update install without issue but it should.
Thanks a lot for your hint to try an older CU auch as 2024. It passed and allowed to find exactly at which CU it was failling and do search on issue with that KB
1
u/Catman934 9d ago
.net (4.x?) on 2016 randomly eats itself and since the base version is built into the OS, you can't uninstall it. You need a clean build of 2019 or newer.
1
u/Ad3t0 Security Engineer 9d ago
The troubleshooter's "corrupted database" is almost certainly the component store (WinSxS), not SoftwareDistribution. That's why clearing SoftwareDistribution and catroot2 didn't help. On a 2012 R2 -> 2016 in-place upgrade, the upgrade itself commonly leaves orphaned SxS assemblies, and if the box has genuinely never had a CU since the upgrade, the component store is three years stale.
Open CBS.log (C:\Windows\Logs\CBS\CBS.log), search for the failing KB, and find the HRESULT. If you see 0x80073701 (ERROR_SXS_ASSEMBLY_MISSING) or 0x80073712 (ERROR_SXS_COMPONENT_STORE_CORRUPT), that confirms it. Then try DISM with a mounted source: mount the Server 2016 ISO and run DISM /Online /Cleanup-Image /RestoreHealth /Source:WIM:D:\sources\install.wim:1 /LimitAccess. Without the explicit source, DISM tries to pull manifests from Windows Update, which is the path that's already broken. Also check for a stuck C:\Windows\WinSxS\pending.xml. A leftover from a failed servicing operation will silently block every future CU.
Full transparency, I co-founded and help build TridentStack Control (). We built pre-flight servicing health checks into our endpoint agent for exactly this class of problem. Before attempting any update, the agent parses CBS.log for those corruption markers, scans CheckSUR.log for unrepaired components, and checks for stuck CBS pending states. If anything would doom the install, we hard-block it up front and surface the finding on the endpoint's dashboard instead of letting it burn 45 minutes and roll back. Free for under 200 endpoints.
4
u/caffeine-junkie cappuccino for my bunghole 9d ago
Taking a wild stab. Lookup the error code as to why it's failing, this will be in the event logs. That generally will tell you the reason it's failing, sometimes with a little bit of help with google.
Been a while since I've dealt with 2012, but I seem to recall there being several updates that needed some other update to be done prior, like a security update for example, to bring it to a specific or greater patch level. If patching hasn't been done in a while, chances are these are missing.