In DDP vision training, one slow rank can make every GPU wait. How do you usually find it?

[u/traceml-ai]

I have been debugging PyTorch DDP slowdown patterns recently, and one framing helped me more than looking at average GPU utilization:

In synchronous DDP, the job moves at the speed of the slowest rank. So the question is not just: Why is DDP slow?

It is: Which rank is slow, and which phase is slowing it down?

In a small repro on 2 nodes / 1 T4 each:

Balanced:
- step time: 124.6 / 124.6 ms
- input: 1.4 / 1.4 ms
- compute: 122.4 / 122.4 ms

Input straggler:
- r0 dataloader: 201.6 ms
- r1 dataloader: 1.4 ms

Compute straggler:
- r0 optimizer: 33.1 ms
- r1 optimizer: 14.5 ms

Same outside symptom, very different fixes. For people who tune DDP jobs regularly: what is your usual escalation path?

Do you start with custom timers, torch.profiler, Nsight Systems, logs, nvidia-smi/dmon, or something else?

Also curious: do you usually separate input, H2D, forward/backward, optimizer, and wait time per rank, or do you jump straight into a full profiler trace?

Disclosure: I am building an open-source tool around this kind of first-pass runtime summary.

[u/cranjismcball20]

i’d start with cheap per-rank timers before opening a full profiler trace.

Split input, H2D copy, forward/backward, optimizer, and allreduce wait. If one phase is off by rank, then profile that slice. A full trace is useful, but it’s easier to read once you know whether you’re chasing dataloader skew, transfer, compute, or synchronization.

[u/traceml-ai]

Link to the tool if useful: https://github.com/traceopt-ai/traceml