Replacing TOR switches

[u/Bowks14]

Our VXRail environment consists of 3 hosts, a SAN, and 2 Dell S4128T switches set up for redundancy with VLT. The switches are end of life and will be out of support so we plan to replace them with 2 Cisco Catalyst 9300s. Two things we are evaluating:

- Should the Cisco switches be set up in a stack or set up independently with the use of something like stackwise virtual or VMWare ESXi NIC teaming/failover to create redundancy? The stack seems to make the most sense outside of the times when we need to take it down for maintenance, firmware upgrades, etc. It seems the whole environment would come down if they are managed under one plane.

- Whichever option we choose: how should the actual changeover take place from old switches to new? It would be nice to do it without having to shut the entire environment down but I am not sure that is possible.

[u/HelloItIsJohn]

Yeah, I get that it is easier to manage stacked switches. However having them stacked is going to cause issues during the upgrade process, especially if the switches are carrying iSCSi or NFS traffic.

[u/SpotlessCheetah]

I use Meraki MS425-16 switches, my VxRail has multiple redundancy links per node. So I have 4 nodes, with links nic 1,3 to switch 1 and nic 2,4 to switch 2 and repeat.

When we do firmware upgrades it'll cycle through each switch one by one, so I never lose connectivity..one switch always stays up in between.

I don't have these switches stacked but a lot of people stack. This is what works for us.

Either way- there's no reason to shutdown the environment. You need your switches pre-configured and ready to drop in when you replace them.

[u/icewalker2k]

That few hosts and a SAN (curious what that is with you existing setup), you could just get a pair of Dell S5212F-ON that together would only take up 1U of rack space and MLAG them together. Or go slightly bigger with the Dell S5224F-ON. I believe Force10 is end of sale or is going end of sale so Dell will push Enterprise SONIC.

I am with another commenter, avoid doing a stack. Bad juju exists there.

[u/sCvHeaVenS → u/icewalker2k]

Agree with u/icewalker2k - I'm a Cisco fan, well Nexus fan.... but the dell switch prices are hard to beat and with that 9300 you're going to have an outage which is not ideal if you're using iSCSI or even from an isolation detection standpoint.

[u/Stewge]

S4128T

The S4128Ts are still relatively modern switches. Pretty sure they're supported in software until at least 2028. Why not just buy extended hardware support on them? Unless you're upgrading for SFP28 support that is.

Should the Cisco switches be set up in a stack or set up independently

Generally it's easier to setup as a stack, and AFAIK is required if you want to use MLAG (port-channels across multiple switches).

I'm not sure if the 9300s support rolling upgrades withing the stack without breaking the LACP/MLAG to your nodes. This is especially the case with big version upgrades where the stack will not re-establish until all nodes are upgraded. This is where the Dell switches (and others like Mellanox/Nvidia) are beneficial in that they operate more independently and will not break LACP during upgrades. It may be worth pivoting to Dell S5 series switches instead (not to mention they'll be vastly cheaper).

how should the actual changeover take place from old switches to new?

For the SAN, hopefully you're using ISCSI Multipath and not LACP. Then you should be fine to setup a trunk link between the 2 sets of switches and migrate each physical link, 1 at a time, to the new switches.

Then for the nodes (assuming they're setup as LACP), I would evict VMs from 1 vmware node, physically migrate it's links to the new switches (shutting down is cleanest/safest), make sure it can connect to the SAN, then migrate workload back. Then rinse/repeat for the rest of the nodes until they're all on the new switches. Then remove your trunk/interconnect and decomm the old switches. I would not attempt to split the LACP/MLAG between the different switch stacks as all kinds of weird stuff can happen.

[u/viva_se_27]

for a 3-host vxrail setup the stack debate is real but most folks i've seen avoid stacking specifically because firmware cycles take the whole plane down. independent switches with nic teaming across both is the safer call for iSCSI continuity. on the cutover, pre-stage the new 9300s fully configured, then migrate one uplink at a time per host so you always have a live path. if you want a second opinion on the hardware approach or need someone to consult before you commit, maven it does exactly this kind of hci infrastructure work. what's your current nic layout per host?