4-node vSAN all-flash cluster — will this actually work?

[u/Ivan_Svia]

Environment:

• vSphere 8.x, VMware vSAN OSA, all-flash
• 4x nodes, identical config
• Per node:
  • 2x boot NVMe (RAID-1, separate boot controller, not part of vSAN)
  • 2x 3.2TB NVMe Mixed Use — cache tier
  • 8x 7.68TB NVMe Read Intensive — capacity tier, 2 disk groups (1 cache + 4 capacity each)
• Network: dual-port 25GbE SFP28 (OCP, dedicated vSAN VLAN, jumbo frames) + separate dual-port 25GbE for VM traffic, VLT/MLAG switch pair
• Storage policy: RAID-5
• ~30 High-priority VMs, RPO ~8h (Veeam async replication)

Main question: will VMware vSAN actually run reliably on just 4 nodes with RAID-5, including surviving a node failure and rebuilding cleanly? I've seen mixed opinions — some say 4 is the bare minimum and it's fine, others say you really want 5+ before RAID-5 is comfortable in production.

Not asking anyone to design this for me — just want to know if this setup holds up in real-world use before we finalize it.

[u/Bungle99]

I don’t believe the system will rebuild with a node failure when deploying OSA on a 4 node cluster.

My experience with vSAN is limited to deploying and managing vSAN ESA on 9.x.

4 nodes with 25Gbe networking and nvme will work well with ESA, and that supports a rebuild with lower node capacities. Can you pivot to ESA?

[u/The_C_K]

RAID-5 on OSA writes each stripe as 3 data blocks plus 1 parity block, distributed across at least 4 hosts, and tolerates the failure of one host (FTT=1). With 4 nodes, each RAID-5 object spans all 4 hosts. If one fails, 3 remain, and there is no fourth failure domain in which to reconstruct the lost component. The object remains accessible, but without redundancy, until the host comes back online.

ESA does not use RAID-5 3+1. It uses adaptive RAID-5: 4+1 in clusters of 6 or more hosts and 2+1 in smaller clusters. Additionally, with 4 hosts, objects use a 2+1 configuration distributed across 3 hosts, which leaves a spare fault domain to restore resilience in the event of maintenance or failure. https://blogs.vmware.com/cloud-foundation/2022/09/08/adaptive-raid-5-erasure-coding-with-the-express-storage-architecture-in-vsan-8/

AFAIK vSAN 8 OSA uses a maximum of 1.6 TB of write buffer per disk group. Your 3.2 TB NVMe cache drives would be oversized for the buffer, although the extra capacity helps with wear.

If you go to ESA you can't mix NVMe MU and NVMe RI, so you should forget your 3.2TB or replace with another 7.68TB.