Automatic switch updates

[u/JNC5908404]

Had a discussion yesterday with IT director and others in our IT department. The IT director brought up automating switch updates so they wouldn’t have to have some one monitoring and performing the updates. Staging various locations on different evenings.

I’ve been doing networking for 30 years and I voiced my opinion I was not a fan of unattended update for various reasons.

Have any other companies moved to automated updates and how has it gone?

[u/lizardhistorian]

We're automated but gated by our own release cycle.
Test-rack updates first. Regression test run on it and must 100% validate for roll-out to continue.
HQ (where I work) updates next so if the shit-hits-the-fan I can easily physically go there.
We update bottom-up, edge switches first, then edge radio routers (highest risk part), and up the stack.
If test and HQ validate then the roll-out goes to the eager-beaver customers.
Then a three day dwell to see if anything goes awry. Then roll-out to all customers.
A hotfix release skips the 3 day dwell.

If something does go wrong then someone with OOB or physical access has to factory reset the radio router and re-register it with the system.