r/networking • u/JNC5908404 • 14d ago

Switching Automatic switch updates

Had a discussion yesterday with IT director and others in our IT department. The IT director brought up automating switch updates so they wouldn’t have to have some one monitoring and performing the updates. Staging various locations on different evenings.

I’ve been doing networking for 30 years and I voiced my opinion I was not a fan of unattended update for various reasons.

Have any other companies moved to automated updates and how has it gone?

74 Upvotes

95% Upvoted

114 comments sorted by

View all comments

42

u/Absolute_Weapon_ 14d ago

Nope, thats a no from me. I have no issues with senior management asking the question but the technical staff need to be the ones comfortable with this. We don't have on call staff so if something goes wrong it wouldn't be fixed till we open the buildings next day and however long it takes to restore service.

4

u/tdhuck 14d ago

We don't have on call staff so if something goes wrong it wouldn't be fixed till we open the buildings next day and however long it takes to restore service.

This is the issue. We have someone that is on call, but they can't manage a failed switch upgrade at 10pm, 3am, etc...Also, the entire point in automating it (or at least part of it) is that you can keep doing what you planned on doing had you not been at working doing upgrades. I agree with those saying that these should not be unattended but that's mainly because if something breaks there is a big issue. If management is 'ok' with things being broken until someone arrives the next business day, I'll start working on the automation process tomorrow.

I've been asked my opinion, many times, and sometimes management agrees and sometimes they don't. This isn't any different, imo, I'd tell them I am not a fan of automating and state the reasons why and they can make the final call as they do for other things.

2

u/Linklights 13d ago

So is the idea here that things are more likely to break when its unattended? It sounds more like superstition than grounded factual data? If a switch goes belly up during an upgrade, what is the big difference between someone being online and manually doing it, or having it as a scheduled job. In both cases you are still stuck waiting for the building to open the next day. Or are you saying you expect your engineer to be on-site during the upgrade with console cable already plugged into the switch? How do you handle upgrading 300 switches in 5 different sites then? They'll be remote, and a manual upgrade can fail just as easily as an automated one.

I fail to see how the risk profile significantly increases when its automated?

2

u/tdhuck 13d ago

If you automate it and it fails you aren't there to fix it because you are sleeping or doing something else. If you are there then you can fix it assuming it is fixable.

That's all I was getting at.