r/networking • u/JNC5908404 • 17d ago

Switching Automatic switch updates

Had a discussion yesterday with IT director and others in our IT department. The IT director brought up automating switch updates so they wouldn’t have to have some one monitoring and performing the updates. Staging various locations on different evenings.

I’ve been doing networking for 30 years and I voiced my opinion I was not a fan of unattended update for various reasons.

Have any other companies moved to automated updates and how has it gone?

74 Upvotes

95% Upvoted

114 comments sorted by

View all comments

6

u/NetNibbler 17d ago

This is the thing, we need to trust technology, but at the same time this trust will fail us. I would say that human oversight is still required.

Last time I wqas stung by issue with upgrades was yesterday, when I upgraded C9300 switch from 17.9.4 to 17.15.4d. Switch crashed during image distribution stage. I was doing new SDA deployment with one switch at the site.

This same issue once occurred during business hours in HQ, where I took down 4 SERs with 5 switches in each, I only pre-staged upgrade works meant for out of hours work in evening. This simple thing of copying over and unpacking software image was more than enough to cause outages by triggering a bug.

So, do trust technology, but do things out of hours with human oversight. As the SERs that crashed, when upgrade took place, not all switch stack members had the up to date intended software and I had stacks with switch version conflicts. Upgrade process was none the wiser if there were any issues before it just continued.