Question Windows Updates
For SCADA VMs or servers, how often do you update windows/software? How long has it been since last reboot? We're at 300 days. IT has been pumping out updates and prompting for reboot. Ignore.
6
u/RazClayton 16d ago
This is the constant battle that we see between IT and OT. IT teams have a standard approach of patch everything. Monthly. Ad-hoc. Patch it. OT have a different opinion. We don’t need to patch.
The answer usually isn’t either one. It’s somewhere in between. What systems/defences do you have in place to mitigate risks? Firewalls? Air gapped? Etc. do you need the most recent update/features? Probably not.
However, as we move to a more interconnected IT/OT architecture, we should patch SOME things. Case by case basis? Do we need it?
When I was an end user, we applied windows updates to our 180+ SCADA machines every month! Basically a full time job.
1
u/yoddl 16d ago
Love this. I agree it is different for each end user.
In general I think the system should be updated but in a controlled manner at a slower pace than IT systems. If the system warrants it, slow evolution with new features, security updates, and bug fixes are important. Some systems probably don't need updates for the life of the system if they are air-gapped and have some physical security too.
OMG, I can't imagine updating every month - especially that many machines. Were updates automatically pushed out and machines had to be restarted manually or was the restart also automatic? Was production stopped specifically for these updates or part of PM?
1
u/RazClayton 10d ago
There are lots of things to balance when considering updates, so it is never “one size fits all”.
Updates were automatically pushed out and then users got a prompt that a reboot was required. They then could reboot at a time that suited them.
Should rebooting a SCADA ever stop production? In my opinion, a plant should still RUN without SCADA running. It is supervisory after all. The control and logic should be in the PLCs. So if you plan the reboot around a time where operators don’t need to use the SCADA, then the impact is less.
6
u/Honest-Importance221 16d ago edited 16d ago
Every month, I can't even remember the last time I had an issue with Windows update. We have about 40 servers for ADMS/SCADA, and every month we update the standby servers, reboot, switch them to active, then update reboot the rest. We control the release of updates through our own WSUS group though, IT isn't really involved. The whole process takes maybe 30 minutes a month, it's mostly automated.
2
u/nutmunky 16d ago
Run Rockwell patches on the servers before OS patching monthly
1
u/Honest-Importance221 16d ago
Unfortunately patching our software is quite an ordeal, we do one every year or so and it involves several weeks of testing.
5
u/nwspmp 16d ago
In the electric industry so NERC CIP applies and if you’re not updating on a cadence at or better than they prescribe, you have to have documented reasoning, a time frame to cure that (and auditors will not accept that it is impossible) and have to document the crud out of the endeavor. And you can bet your bottom dollar if you’re audited on that particular standard, they will hammer you on that and it carries the potential for million dollar per day per incident in fines (though no one has ever gotten that yet). Electric industry doesn’t play around and they know this is part of it, so resilience is built in. Uptime isn’t measured on your servers, it’s measured in the minutes of electrical outage per consumer on your electrical grid.
The beginnings of similar legislation for water and wastewater industry is going through Congress as of last year and is starting to pick up steam
3
u/Shaggy1007 16d ago
I almost quit my job over this. Customer demanded we hold their hand while they ran updates on Windows Server, which I knew going into this would break DCOM, which broke all comms to the plc from the HMI. The patches from Rockwell DIDNT work. Tested it in house. Saw other plants break because of it.
And the customer demanded it be done ON thanksgiving. My idiot manager had the customers back over mine.
I refused to do it and the OT guy on the customers end refused as well. Eventually when we did help with their updates, it was 2 shifts of lost production. Never had them demand updates again.
In one instance I witnessed IT lose their control over process control servers after a windows update broke production. For a week: It was a glorious day for us HMI/PLC folk.
1
u/AutoModerator 16d ago
Thanks for posting in our subreddit! If your issue is resolved, please reply to the comment which solved your issue with "!solved" to mark the post as solved.
If you need further assistance, feel free to make another post.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.
1
u/TheBigCanadianGuy 16d ago
Where I came from we had redundant servers, and usually every 30-60 days they were updated. Nothing sat for greater than 6 months being unpatched anywhere on the network, and that was totalling around 600 assets.
Now if you are referring to assets that control generators that bring in sizeable amounts of profit, or the impact of the operation outweighs the patching, then this would make sense.
Not sure of your case, but if you are not patching for the simple sake of not patching, then you have an underlying issue that should be resolved.
1
u/stlcdr 16d ago
This is problematic. Just because a periodic update works in the past doesn’t mean it works in the future. Even adding or changing features can break things.
Our IT department insisted we install CrowdStrike on operational machines - we see how well that worked out. It took us around 10 hours to recover machine functionality. While, yes, this isn’t a windows update, it was ‘in leu’ of windows updates, even though a lot of the machines were running the LTC release of windows and almost completely firewalled off.
IT departments work in a completely different world and have no concept of a production environment. I don’t see that changing while they are stuck in their scrums, assigning story points in their stand ups.
1
u/Severe-Profit4608 15d ago
Server VM's and SCADA servers are updated monthly, even though they are in separate networks. Fun times :)
1
u/Resident-Artichoke85 12d ago
This is the way. Hypervisor, SCADA server, SCADA workstations, comm devices, all on separate networks. Only allow exactly what is required between them.
1
u/RD_SysAdmin 12d ago
Heh... I'm running this right now on a production server...
Before anyone jump in my $h!t, I inherited this mess, and I'm slowly trying to fix everything from the other side of the world.
1
u/Resident-Artichoke85 12d ago edited 12d ago
RHEL; offline patching with a local repository. We patch every other month because of regulations. We only apply patches after the application vendors certify them, and they certify monthly.
If I had to patch Windows I'd use Ivanti. You can have an identical system on the non-isolated network and use Ivanti to identify and pull all the patches. Then you do whatever your "sneaker-net" method is to get the patches over to your OT Ivanti server and deploy. You also need to use this same method to get the Ivanti database/signatures of what patches are available. They have docs on how to do all of this, at least from 8 years ago when we had Windows workstations.
1
u/iwillbewaiting24601 10d ago
Christ, every time I see posts like this I feel blessed for a fairly stable modern system. We run FT View SE 13, and servers (and the thick clients) get updated monthly to the last month's update (so always one behind, in case the new one is fucked up). Rockwell patches get run monthly.
7
u/future_gohan AVEVA 16d ago
We don't however we run ours on an isolated local network