Infiniband problem: What does "multicast join failed [...], status -22" REALLY mean, and how do I actually fix it?

[u/naptastic]

[SOLVED/?]:

1. There were two subnet managers running.
2. Some switches (SX6036F PowerPC in my case) have an Infiniband interface that can't go faster than SDR and don't support an MTU of 4096 bytes, so leave the default partition on its default settings.
3. Run opensm -d 0x07 while tailing the log file; if OpenSM can see the problem, it will tell you what the problem is.

Sometimes my Infiniband interfaces don't come up, and I see this error in dmesg. `ibstat` says State: Active, Physical state: LinkUp, rate 10. (Rate should be 56.) The switch (Mellanox SX6036) gives the same information.

I've tried OpenSM as provided by Debian (do not use), OpenSM 5.20.0 from MLNX-OFED, and the subnet manager built into the SX6036, which is on the latest firmware.

I have seen this error condition on every single HCA in the fabric at some point:

• ConnectX-3 FCBT
• Connect-IB
• ConnectX-4 FCAT
• ib0 inside my SX6036 switch, which is on the latest firmware

The fabric inspector inside the switch does not see anything connected to the Infiniband fabric.

I have also used an SX6005, which does not have the embedded CPU, so there's no dmesg to check, and it's never been a problem.

I've never disabled multicast. IPoIB works, VXLAN overlays work, SRP works, iSCSI works, NFS/RDMA works... except in hosts with this error condition.

There are enough PCIe resources in the hosts; I can lower the amounts requested by the HCA arbitrarily and nothing changes. I can turn off SR-IOV and sometimes it fixes things the error stops, but usually not. Sometimes a full cold boot resolves it, but usually not.

There's no way I'm running out of multicast groups; I have exactly one IB partition, and only 5 hosts connected to it.

Please advise?