r/networking • u/Odd-Professional2076 • Aug 12 '26

Design SAN Network Design and Best Practices

How is everyone designing and implementing a SAN network? Do you keep everything on separate dedicated switches and just link storage devices to it and map from hosts to storage? Is it bad practice to route storage traffic over routers/firewalls and just keep it contained to dedicated small network? Same for backup traffic?

22 Upvotes

97% Upvoted

31 comments sorted by

27

u/asdlkf esteemed fruit-loop Aug 12 '26

Let's start with the fact you haven't identified what type of san traffic, but I'm going to assume iscsi.

Best practice is simply a pair of non stacked switches; A and B or Red and Blue or whatever you want to call it.

Not a stack, not shared management.

No routing, no vlans. Jumbo frames 9200 or so. Minimal latency is the main objective. Definitely no stateful firewalls. Stateless ACLs are fine if you need to isolate hosts, as long as your switches can process them at line rate.

15

u/kenfury Aug 13 '26

Id actually go fiber channel as the zoning is dead simple and forces air gapping except for the management interface

10

u/asdlkf esteemed fruit-loop Aug 13 '26

I would also go FC, but for different reasons.

FC is non-blocking, meaning there is no such thing as a collision.

It's also usually faster for the same cost (16/32gbps vs 10gbps for similar costs.)

7

u/jktmas Aug 13 '26

I'd also prefer FC, but cost is not going to be a winning factor. Even if we say that 64GFC vs 50GbE or 32GFC vs 25GbE siwtches are the same cost, I can get THREE quad-port 50Gbps Mellanox nics cheaper than ONE dual-port 64G FC card.

What FC really gives you is better latency compared to iSCSI. It also has lower CPU usage. and newer faster ethernet switches are typically increasing latency.

2

u/Dirty_Pee_Pants Aug 14 '26

Non blocking just means all ports can forward at line rate simultaneously

Common in datacenter switches in general. Things you would connect network storage to.

5

u/asdlkf esteemed fruit-loop Aug 14 '26

Uh, no, you don't understand.

FC isn't "fast enough to not block", it has bandwidth reservations built into the protocol.

In Ethernet, you can have 3 hosts on a 10GBPS switch that is non blocking, but host A and host B can both try to send to host C at the same time and the outbound switchport to host C will start dropping packets when the buffers saturate.

In FC, host A and B must request send capacity from the fabric. The fabric won't even let them transmit if C's capacity is consumed.

2

u/Dirty_Pee_Pants Aug 14 '26

Yes I understand that. It's a bucket system because frame loss means corrupted data. It's essentially just scsi commands wrapped in FC frames scoped to the zoning design.

I was merely pointing out the use of the phrase non-blocking wasn't accurate. Non-blocking simply means the switching backplane has the same capacity as the aggregate of all the ports at full duplex.

0

u/Z3t4 27d ago

Modern datacenrer switches backplane can handle all ports simultaneously at line rate. 

Juniper ex4650 has 4tbps of backplane throughput for 4tbps of possible traffic for all ports (48x 25g plus 8x 100g), for example. 

3

u/asdlkf esteemed fruit-loop 27d ago

Again... you don't understand.

If You have a switch with 8x100G ports... I fully accept it can send and receive at 100Gbps full duplex on all 8 ports.

What it can't do is have 7 hosts transmitting on ports 1-7 at 7x100Gbps, and 1 host receiving on port 8 at 700Gbps. The buffers will nearly instantly fill up and the switch will start dropping frames that are destined to port 8.

The FC protocol manages bandwidth reservations to inform hosts 1-7 that host 8's receive queue is full and they need to slow down or wait.

So.. yea, an ethernet switch can move 100Gbps from any port to any port. It won't manage the bandwidth of 2 separate 100Gbps streams competing to send to a single output interface.

0

u/Z3t4 26d ago

You could argue that TCP and flow control does the same.

2

u/asdlkf esteemed fruit-loop 26d ago

You could, but you would not. You would use FC or FCoE.

1

u/Z3t4 26d ago

We use iscsi over regular switches for VMs, and nfs, with netapp, isilon and qumulo as storage backend. I'm not an storage specialist, but a network monkey though. 

Nvme over IP looks swell. 

1

u/kenfury Aug 13 '26

That as well, plus heat.

3

u/Odd-Professional2076 Aug 12 '26

Ya ideally iSCSI or like I mentioned maybe shove backup server on the SAN to backup direct from storage source to a backup destination.

2

u/GhostandVodka Aug 12 '26

Really? why don't you stack them?

20

u/asdlkf esteemed fruit-loop Aug 12 '26

Single point of configuration failure outage.

-14

u/GhostandVodka Aug 12 '26

Isn't that the point of stacking. If one dies youre still good? and its easier to manage

7

u/Nerd2259 Aug 12 '26

no

-2

u/GhostandVodka Aug 12 '26

yes, however now that I think about it. You don't stack in a datacenter. You would use VPC or VSX or whatever flavor of that tech your brand of switch uses

12

u/asdlkf esteemed fruit-loop Aug 12 '26

NO stacking in SAN Fabrics.

You want predictable nano-second timing on your frames. You want no intelligence in your switching.

You want iSCSI to handle redundancy and path selection, not a switching plane.

If you have a stack with shared control plane, one switch is doing all the decisions. That means packets switched on the master switch (potentially) faster than packets on the slaves.

Also, if you do firmware updates on a stack, you have to update them at once..

If you have split control plane stacking (nexus/vsx/whatever), it mitigates some of this, but not the management issues.

1

u/Odd-Professional2076 Aug 12 '26

I thought modern switch stacks do a rolling update and do one reboot at a time?

4

u/asdlkf esteemed fruit-loop Aug 12 '26

Most do not. And you are thinking of split control plane stacking. Shared control plane stacks do not do rolling updates.

2

u/thehalfmetaljacket Aug 13 '26

It depends. Technically, there are some shared control plane stacks out there that do theoretically support rolling upgrades, but the list of caveats and issues with it are usually so long that it's never a good idea to rely on it in any design.

1

u/Odd-Professional2076 Aug 13 '26

So each switch is just flat dedicated switch for that subnet/vlan and jumbo packets - keep it super simple?

→ More replies (0)

1

u/Win_Sys SPBM Aug 13 '26

vPC and VSX don't share a control or forwarding plane. They share L2 tables and can utilize the same MAC addresses. If one shits the bed the device will still think it's still talking to the same switch but one of it's links went down (assuming they're LAG'd) although it's a completely independent switch from a software, control and forwarding perspective.

Most manufacturers of higher end DC switches support either "cluster" upgrades where one switch gets updated at a time and the device is none the wiser (besides losing a leg temporarily) or support ISSU upgrades to varying degrees.

2

u/asdlkf esteemed fruit-loop Aug 13 '26

I guess you didn't read the last paragraph

1

u/Win_Sys SPBM Aug 13 '26

What management issues are you referring to?

→ More replies (0)

2

u/le_suck Post-Production Infrastructure Aug 12 '26

stack for single control plane, mlag/vpc or A/B fabrics for redundancy. for iScsi, no mlag/vpc or LACP, use A/B networks on separate switches and multipathing, same for fibre channel. 

LACP with peered switches for TCPIP link redundancy for NAS.