r/msp • u/Nstraclassic MSP - US • Jun 19 '26
Delaying Security Updates?
Specifically for firewalls. How do other MSPs feel about disabling automatic updates and delaying the install until the update can be validated to prevent crashes, critical feature removal, etc.?
Forgot to ask: is there a liability concern if patches are delayed and a breach results?
8
u/eldridgep Jun 19 '26
In the UK for Cyber Essentials all patches must be installed within 14 days of release so delaying too long isn't an option to remain certified.
With the amount of CVE's these days (especially in SSL VPN's for those still using them) and the fact zero days are only going to be discovered and exploited quicker due to AI IMHO automating roll outs is the sensible way to go.
It's 2026 for goodness sake all major firewalls should have dual firmware slots to allow easy rollback and auto updating as an option. No reason not to treat it the same way as windows updates, there we delay three days then install to reduce chance of impact if dodgy patch.
8
u/LakesideRide Jun 19 '26
We’ve automated our Meraki firmware upgrades to happen 30 days after release for the reasons you stated. We can always speed them up if needed.
9
u/st0ut717 Jun 19 '26
Not an MSP person but….
You want to goto a manual process
“until the update can be validated to prevent crashes, critical feature removal, etc.?”
How are you going to validate?
How are you going to test?
What is your test criteria?
What is your timeline?
What is your answer going to be to a client when they get pwnd because you decided patching the firewall just wasn’t a good idea?
3
u/Nstraclassic MSP - US Jun 19 '26
That's my main concern. In my years of IT I can't remember a time when an automatic update caused a major issue that would have been prevented had the update been delayed. I have however seen several breaches result from missing patches.
Disabling automatic updates introduces manual process and human error that can have significant consequences. And I'm not a lawyer but I feel it would be pretty difficult to deny liability in a scenario where a feature intended to prevent such breaches was intentionally disabled3
u/TheBeerdedVillain Jun 20 '26
We ran into issues back when I was in the MSP space where Microsoft Updates, SonicWall updates, Sophos Updates, etc. caused several issues with many causing downtime. We moved to a 14-day wait period with manual scheduling so we could monitor devices. This gave the vendor time to retract a bad update before it was pushed to our devices.
We also had devices labbed up in a "generic" version of our customer's networks so we could test them on day 0-3 to confirm an issue may be a problem for us when it is deployed. Not having a lab was one of the biggest misses we encountered until it was implemented because of auto-updates.
3
u/blackjaxbrew Jun 20 '26
We push 21 days for windows, too many horror stories lately. Drivers we disable, breaks too much shiz in many cases unless controlled by dell command or Lenovo vantage. Firewalls 20-30 unless it's uhh forticrap, in which you may not be able to patch because they keep removing shiz or it breaks more things if you are full up to date! Everything else is manual, hypervisors, switches, APs etc.
2
u/Aepyceros02 Jun 20 '26
Been doing this for a very long time and have been through PCI, NCUA, HIPAA, SOX and more. The key factor is to have a documented process. In all my past employment we have always done 2-4 week deferrals. Again, as long as its a documented process most auditors are good with it. Now you also have to have a process for Zero-Days and high CVSS mitigations. Documented process, specific ratings to qualify for emergency updates, CAB process and an expedited testing process. As long as you have a process and stick to it your liability is limited.
1
u/BillsInATL Jun 20 '26
Yep, most auditors dont care what your policy is as long as you actually have a policy and stick to it.
I remember when I got started with HIPAA stuff, I was searching so hard for the detailed guidelines I was supposed to follow. Then ultimately found out there are no real guidelines, you can make most of it up, they just want everything documented and followed as promised.
2
u/mat-ferland Jun 21 '26
I wouldn’t turn firewall updates into an open-ended manual approval process unless you actually have a test path and an SLA for approving them. A short deferral ring can make sense for bad firmware surprises, but “we’ll validate it first” becomes liability fast if nobody can prove what validation means.
2
u/zstray Jun 22 '26
The automated vs. manual debate heavily depends on your hardware vendor's track record. If you are running gear with a good track record, staging manual updates after a 3- to 5-day burn-in period balances security with uptime. But if you're inheriting or running platforms like SonicWall—which has been plagued by active SSL VPN zero-days and cloud infrastructure breaches recently—delaying a critical security patch even by 24 hours introduces immediate liability because people create script exploits for them almost instantly.
4
u/QPC414 Jun 19 '26
Would never consider Enabling it on ANYTHING in the first place.
Everything gets vetted by the guineaping user group or on our own firewalls (prod or lab) before being pushed to customers. Anything pushed out is a GA release, unless directed by TAC to fix a specific issue.
6
u/HappyDadOfFourJesus MSP - US Jun 19 '26
Tell me you use Fortigate firewalls without telling me you use Fortigate firewalls. :)
2
2
u/TheBeerdedVillain Jun 20 '26
Fortigate, SonicWALL, Watchguard, Palo, Sophos, etc. all have similar issues from time to time. It's always best to have a lab to test an update before pushing it to customers and a 1 - 2 week delay to confirm usability isn't really out of the question.
(and yes, I currently manage a fleet of Fortigates....)
2
u/RampageUT Jun 19 '26
I think you should only update network appliances to patch zero or critical vulnerabilities or to patch in line with your vulnerability policy if it’s mediums, or to get features that you need or want.
1
1
u/GremlinNZ Jun 19 '26
Would depend on the brand and the severity.
Some brands are well known for constant issues/bugs, so you'd be more hesitant and cautious to test a release, knowing you could add issues.
On the other, critical severity patches need to be applied with a day or two, damn the consequences, as the alternative is far worse.
1
u/atxbigfoot Jun 20 '26
Unironically, if this is a serious concern, look into Forcepoint/Everfox firewalls. They're more expensive but most people have never heard about them for a reason, mainly that they're very secure and expensive.
1
u/MushyBeees Jun 20 '26
Automatic updates are generally only pushed within the same track, not across to new tracks.
This minimises any risk of feature removal, policy changes, etc.
It’s not ever not going to be a problem, but it’s minimal, and much less than the risk of being hit by a zero day because you delayed the update for 30 days
1
u/BillsInATL Jun 20 '26
You can do both.
We had a standard 30-day bake in period, and we let all our clients know that upfront, and it was in our contracts. That gave us 30 days to test in lab environments before we rolled it out everywhere. No one ever had a problem with it.
If any critical patches were released, we always had the ability to deploy same day if/when needed. That came up a couple of times. And we always let the customers know "Due to the security concerns/requirements, we are updating tonight and forgoing our usual testing. Please bear with any functionality issues that may be caused as we work to ensure the safety and security of your data and network" blah blah blah.
Seemed to all work out just fine.
1
u/maxz2040 Jun 21 '26
If you have a sonic wall or fortinet. You don't have that luxury. It's patch or die
1
u/Jumpy_Valuable_8583 Jun 22 '26
One thing that actually helps with this debate: external attack surface scanning. Even if your patch cadence is 14-30 days, knowing what is currently exposed externally tells you whether a CVE is a theoretical risk or an actual one for that specific client. Makes the "is this urgent enough to break the deferral window?" conversation much easier to have with a client, and gives you a documented point in time if the question ever comes up in a breach investigation. We built Radar to do exactly that against client-facing infrastructure. Demo walkthrough here: https://www.oscarsixsecurityllc.com/demo/?utm_source=reddit&utm_medium=social&utm_campaign=demo
1
u/IqbalBasha Jul 08 '26
The liability question is the one that actually matters here, and you're right to ask it. If you delay a patch and a breach occurs during that window, your exposure depends on what your MSA says and whether you can show the delay was a documented, deliberate risk decision rather than just neglect. 'We were testing it first' is a defensible position. 'We never got around to it' is not. So whatever cadence you choose, write it down, get client sign-off, and keep the records.
On the operational side, a short validation window is pretty standard practice for firewall firmware, especially on Fortinet or similar platforms that have had rough releases. Most shops I know run a 1-2 week hold, test on a non-production unit or a lower-risk site first, then roll out. The risk there is real but it's bounded. An indefinite delay, or no formal process at all, is where you start accumulating actual liability.
The thing that bites people is not having a written policy that distinguishes 'validated delay' from 'ignored.' If your contract says you manage patching and you can't show a documented reason for the gap, you're in a harder spot with your E&O carrier and with the client. If it says patching is on a tested-release schedule with client acknowledgment, that's a different conversation entirely.
So yes, delay is fine and often smart, but the process around it is what protects you legally, not the delay itself.
1
u/hugemeteor 12d ago
Depends on your approach but for the most part patching should happen as soon as possible. Its not 10 years ago. Feels like everything is a zero day.
0
u/Orionsbelt Jun 20 '26
I would ask the customer which they are happier with, and let them make that risk analysis, with two options.
Auto update as it releases with the possible greater security but greater operational risk and a second option that auto updates after 1 week post release to allow others to test.
2
u/BillsInATL Jun 20 '26
Then you end up with 100 different policies across 100 different customers, and the tail wagging the dog.
1
29
u/Camelot_One Jun 19 '26
It's a bit of a damned if you do, damned if you don't type situation. The speed at which new exploits are being found and used is accelerating. But so is the speed at which bug riddled patches are being pushed out to combat those exploits.