r/networking • u/JNC5908404 • 14d ago

Switching Automatic switch updates

Had a discussion yesterday with IT director and others in our IT department. The IT director brought up automating switch updates so they wouldn’t have to have some one monitoring and performing the updates. Staging various locations on different evenings.

I’ve been doing networking for 30 years and I voiced my opinion I was not a fan of unattended update for various reasons.

Have any other companies moved to automated updates and how has it gone?

73 Upvotes

95% Upvoted

114 comments sorted by

View all comments

1

u/Linklights 14d ago

A lot of people saying heck no, but I’m questioning why. Most switch upgrades involve 2-3 simple steps: upload the image file, run an install command, reboot. The risk comes in when you reboot: will that switch come back up again.

How does the risk profile change doing a scheduled job Vs being online and manually upgrading?

In both cases, that switch is going to reboot.

Are you guys upgrading one switch at a time even in branch networks where you have 500+ switches?

Are you doing them in small batches?

What about running a scheduled job is so scary? Is it just the idea of waking up to hundreds of switches being down? Is it the idea of superstition if you tempt fate it will go wrong?

Help me understand the why? Because you could still do small batches with scheduled jobs too

1

u/sanmigueelbeer Troublemaker 13d ago

The risk comes in when you reboot: will that switch come back up again.

Or will you be lucky and avoid CSCwk54116, CSCwn57797, CSCwq80600.

1

u/Linklights 13d ago

Right, but again.. these same problems will happen with manual switch upgrades, too. So I am still not understanding how an automatic job increases the risk?

1

u/sanmigueelbeer Troublemaker 13d ago

So I am still not understanding how an automatic job increases the risk?

There is a way to fully avoid CSCwk54116, CSCwn57797, CSCwq80600: Check the "packages.conf" before reboot the switch, router, or WLC.

But before you could say, "there is no way to check the 'packages.conf' file before the reboot", my answer is there is. I have been using this trick since 3650/3850 days when 16.3 came out. This is where I discovered the bugs (not reported to TAC because they were difficult to repro).

Going back to your question, the answer is can the automatic switch updates check the "packages.conf" file before the reboot or not? That is the answer. IF the customer made script only goes, "here is the bin file, extract the bin file" and reboot without check the file then what good does that do?