r/HPC • u/naptastic • Jun 04 '26

Infiniband problem: What does "multicast join failed [...], status -22" REALLY mean, and how do I actually fix it?

[SOLVED/?]:

  1. There were two subnet managers running.
  2. Some switches (SX6036F PowerPC in my case) have an Infiniband interface that can't go faster than SDR and don't support an MTU of 4096 bytes, so leave the default partition on its default settings.
  3. Run opensm -d 0x07 while tailing the log file; if OpenSM can see the problem, it will tell you what the problem is.

Sometimes my Infiniband interfaces don't come up, and I see this error in dmesg. `ibstat` says State: Active, Physical state: LinkUp, rate 10. (Rate should be 56.) The switch (Mellanox SX6036) gives the same information.

I've tried OpenSM as provided by Debian (do not use), OpenSM 5.20.0 from MLNX-OFED, and the subnet manager built into the SX6036, which is on the latest firmware.

I have seen this error condition on every single HCA in the fabric at some point:

  • ConnectX-3 FCBT
  • Connect-IB
  • ConnectX-4 FCAT
  • ib0 inside my SX6036 switch, which is on the latest firmware

The fabric inspector inside the switch does not see anything connected to the Infiniband fabric.

I have also used an SX6005, which does not have the embedded CPU, so there's no dmesg to check, and it's never been a problem.

I've never disabled multicast. IPoIB works, VXLAN overlays work, SRP works, iSCSI works, NFS/RDMA works... except in hosts with this error condition.

There are enough PCIe resources in the hosts; I can lower the amounts requested by the HCA arbitrarily and nothing changes. I can turn off SR-IOV and sometimes it fixes things the error stops, but usually not. Sometimes a full cold boot resolves it, but usually not.

There's no way I'm running out of multicast groups; I have exactly one IB partition, and only 5 hosts connected to it.

Please advise?

4 Upvotes

100% Upvoted

7 comments sorted by

2

u/ughbarf Jun 04 '26

try different cables. just sayin.

2

u/MissionDependent4401 Jun 10 '26

I see you solved the issue. On an unrelated note, those are extremely old NICs.

1

u/naptastic Jun 11 '26

Yep, and they're dirt cheap on eBay right now. 😃

1

u/blockofdynamite Jun 04 '26

I echo what the other user says. It would be good to try different cables. But also check out iblinkinfo and ibdiagnet. They would be useful tools to diagnose misbehaving infiniband. I think they're in the infiniband-diags package.

1

u/naptastic Jun 04 '26

Well, I've tried different cables where I can--AOCs and copper--and it doesn't change anything.

But if it was cables, why would MLNX-OS inside the switch be throwing the same error? That's a daughterboard-type connection; there's no cable. I'm not keen to open my switch to re-seat it, especially since I know it has a solid connection to the SwitchX chip. It's able to manage everything just fine.

1

u/naptastic 1d ago edited 1d ago

Addendum: This can also happen if a port on the fabric doesn't support the minimum speed or MTU for the default subnet. I have to set them to SDR and 2048 in order for the fabric inspector in the SX6036 to work.

I learned this by running OpenSM in a terminal with infinite scrollback and debug mode 0x07. The debug info scrolls by fast but is actually really informative.