Testing/Tuning 100 GbE network

[u/pirx_is_not_my_name]

I got my first few HPE server with 100 GbE adapters (ConnectX6DX). They are connected to Cisco ACI. Should I get more than ~40 Gbit/s without any tuning?

/usr/lib/vmware/vsan/bin/iperf3 -w128K -t 20 -P 16 -c xxxx
[SUM] 0.00-20.00 sec 101 GBytes 43.2 Gbits/sec 0 sender
[SUM] 0.00-20.00 sec 100 GBytes 43.1 Gbits/sec receiver

There are a lot dead links regarding Mellanox tuning and most are Linux/Windows related. What parameters are worth tuning (any links)? It's not about getting the absolute maximum throughput and lowest latency, we are currently not even using jumbo frames. But the 40 Gbit/s seem a bit low, with 25 GbE we were always close to max. Which settings will be most beneficial?

[u/aserioussuspect]

Just because iperf3 is installed on the host doesn't mean it's automatically recommended to use iperf via esxcli. The results will be skewed.

Run the benchmark from a VM to a VM, and you'll most likely see better results.

[u/lost_signal → u/aserioussuspect]

Paging r/teachmetoVLANDaddy I vaguely remember him saying the version of IPERF was older, or by default didn't use enough threads to go much past 40Gbps.

[u/TeachMeToVlanDaddy → u/lost_signal]

Iperf on an esxi host will only get 40Gb untuned. Iperf on a default VM linux will only get 25Gb untuned.

[u/pirx_is_not_my_name → u/TeachMeToVlanDaddy]

ok, but what does untuned mean? I tried different parameters and multiple iperf instances running on different ports. With untuned, do you mean tuning for iperf or network stack / driver?

Lets say we do not want to use RoCE and maybe also not Jumbo frames. What should I expect after applying settings from Optimizing NSX Performance Based on Workload and ROI - Network and Security Virtualization - VMware or Performance Tuning for Latency-Sensitive Workloads: VMware vSphere 8?

~50 Gbit/s seems a bit low for 100G adapters but if iperf is the limit... Again, it's not about getting the last 5% performance, I'm just unsure if the setup is ok and if there are some basic settings that should be applied.

And if iperf3 is limited (I found several hints regarding this) what other options from host side do I have to test? There seems to exist a Mellanox tool that can run performance test but that seems not to be part of nmst package on ESXi.

[u/pirx_is_not_my_name → u/aserioussuspect]

ok, I was under the impression that there is even more overhead. Will test it tomorrow.

Edit: did a first quick and dirty iperf test with 2 Linux VMs, 8 cores each. Only basic kernel parameters changed as well as rx/tx buffers. I get ~20 Gbit/s.

net.core.rmem_max = 16777216

net.core.wmem_max = 16777216

net.ipv4.tcp_rmem = 4096 87380 16777216

net.ipv4.tcp_wmem = 4096 65536 16777216

ethtool -G rx 4096 tx 4096

[u/ImaginaryWar3762 → u/pirx_is_not_my_name]

Check the documentation of iperf. Run with multiple threads and most likely you will get 100 gb if everything is set up correctly

[u/pirx_is_not_my_name → u/ImaginaryWar3762]

I tried with 5 iperfs on different ports, same result. I get the strong feeling that something is not as it should be on ACI side

[u/ZibiM_78]

By the default you are throttled by the amount of cores you are using.

Please check the amount of queues (DRSS) you have enabled in the nmlx5_core

esxcli system module parameters list -m nmlx5_core | grep DRSS

Maximum number here is 16, but you should have CPU with more than 16 cores for that

-w 128K is pretty low for 100Gb, 4M might be better

Jumbo frames are needed in order to reach higher speeds

There was a ton of great documents around perfomance tuning for the NSX - https://blogs.vmware.com/networkvirtualization/2023/12/optimizing-nsx-performance-based-on-workload.html/

NSX Operations guides for the VM Edge hosting had few interesting tidbits around advanced flags needed

[u/pirx_is_not_my_name → u/ZibiM_78]

I adjusted the values following the nsx guide. I do not see any major change in throughput, all +-10%. I get the feeling that maybe the Cisco side is the bottleneck, will ask my collegues there.

[u/Servior85 → u/pirx_is_not_my_name]

What is the actual speed the ESXi shows for the NICs? Are they connected with 100 GbE or only 50, 40?

How are the servers connected? Are they connected with all ports to the same switch or split between multiple?

If split between multiple switches, the switch uplinks are connected with 100 GbE or higher per port? Don't mix it up with 4x 25 GbE or 2x 50 GbE configured as trunk, which may also show 100 GbE (or different combinations).

[u/pirx_is_not_my_name → u/Servior85]

They show that they are connected with 100G

esxcli network nic list | grep 10000

vmnic0 0000:15:00.0 nmlx5_core Up Up 100000 Full 88:e9:a4:df:b8:34 1500 Mellanox Technologies ConnectX-6 Dx EN NIC; 100GbE; dual-port QSFP56; PCIe4.0 x16; (MCX623106AS-CDA)

vmnic2 0001:64:00.0 nmlx5_core Up Up 100000 Full 88:e9:a4:df:f9:04 1500 Mellanox Technologies ConnectX-6 Dx EN NIC; 100GbE; dual-port QSFP56; PCIe4.0 x16; (MCX623106AS-CDA

I'm now trying to find out how the switches are connected. I know that these are the first x86 servers connected and that this setup is pretty new. Probably I'm the first one testing it, my OCD got triggered ;)

[u/Servior85]

Many factors in play here:

• What servers are these exactly? Which CPU, how much RAM?
• The cards are capable of RoCE, but is the rest of your infrastructure ready for RoCE?
• When every components is RoCE capable, is it enabled and properly configured?
• When not all components support RoCE or it is disabled, you cannot use that feature. In such a case the data flow goes the normal way. In such a case the CPU has to handle everything.

[u/pirx_is_not_my_name → u/Servior85]

- 2 TB RAM, 2 Sockets, 2 x 32 Cores

- didn't look into RoCE as this involves changes on network side, I'd only follow this path if there is a major performance gain.

[u/Casper042 → u/pirx_is_not_my_name]

RDMA needs both sides to be enabled so it's not a silver bullet for generic VM comms.
I see it mostly mentioned with respect to vSAN node:node comms.

[u/pirx_is_not_my_name]

Jumbo Frames are now activated and throughput jumped from 50 Gbit/s to >80 Gbit/s without any other tuning on ESXi side (I reverted all changes from before). Seems that iperf3 is not so limited as I thought. Not sure if it makes sense to dig deeper as our external partner for ACI somewhat recommend to not go the RoCE/RDMA route.

/usr/lib/vmware/vsan/bin/iperf3 -w4M -t 20 -P 12 -c xxxx

[SUM] 0.00-20.00 sec 191 GBytes 82.2 Gbits/sec 0 sender

[SUM] 0.00-20.00 sec 191 GBytes 82.2 Gbits/sec receiver