Multi-node training over InfiniBand: a practical NCCL tuning guide

Why this matters in dollars

A 16-node training job on Reviosa runs $293/hr (16 × $18.32). If your all-reduce is 10% slower than the fabric allows, you are burning roughly $29/hr — over $21,000 a month — on nothing. NCCL's defaults are good, but "good" across every possible topology is not "optimal" on yours. The difference between an untuned and a tuned multi-node setup is routinely 10-25% of end-to-end throughput, and the tuning takes an afternoon.

Know the fabric you're tuning for

Our H100 clusters are rail-optimized: each GPU has a dedicated 400 Gb/s NDR InfiniBand HCA — eight rails per node, 3.2 Tb/s of node egress — with GPUDirect RDMA so transfers bypass host memory entirely. Rail-optimized means GPU i on every node connects to the same leaf switch, so the dominant NCCL ring pattern never crosses more switch hops than it must. Knowing this shapes everything below: you want NCCL using all eight HCAs, pinned correctly, with enough channels to saturate them.

Establish a baseline before you train

Never debug communication performance through your training loss curve. Build nccl-tests and measure the collective you actually use — all-reduce for DDP and FSDP gradient sync:

export NCCL_DEBUG=INFO
export NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7
export NCCL_SOCKET_IFNAME=eth0
export NCCL_IB_QPS_PER_CONNECTION=4
export NCCL_MIN_NCHANNELS=32

mpirun -np 32 -H node1:8,node2:8,node3:8,node4:8   ./build/all_reduce_perf -b 512M -e 8G -f 2 -g 1

On a healthy 4-node H100 cluster you should see bus bandwidth in the 350-380 GB/s range at 1 GiB+ message sizes. If you are seeing 200, stop — something structural is wrong (a rail down, traffic falling back to TCP, or HCA affinity mismatched to GPUs) and no framework flag will fix it.

The settings that actually matter

• NCCL_IB_HCA — pin the exact device list. Auto-detection occasionally picks up the Ethernet RoCE device or storage NICs; an explicit list removes the guesswork.
• NCCL_IB_QPS_PER_CONNECTION=4 — multiple queue pairs per connection improves link utilization on NDR fabrics, worth 5-10% at large message sizes in our testing.
• NCCL_MIN_NCHANNELS=32 — more channels means more parallelism across the eight rails. Watch the NCCL_DEBUG=INFO output to confirm channels actually spread across HCAs.
• NCCL_ALGO — leave it alone at first. NCCL's tuner picks Ring vs Tree per message size sensibly; only override after your baseline proves a specific size regressed.
• NCCL_IB_TIMEOUT=22 — raise from the default on jobs above 8 nodes to survive transient congestion instead of aborting a $300/hr run.

Framework-level overlap

Fabric bandwidth is necessary, not sufficient — you also need communication hidden behind compute. For DDP, set gradient_as_bucket_view=True and raise bucket_cap_mb to 100-200 so all-reduce launches cover fewer, larger transfers. For FSDP, forward_prefetch=True and backward prefetching keep parameter all-gathers ahead of the compute that needs them. Then profile: in a trace, exposed (non-overlapped) communication on a well-tuned job should be under 10% of step time.

Two adjacent culprits masquerade as network problems often enough to check every time. A dataloader that stalls one rank stalls the collective for all of them — bump worker counts and prefetch depth before blaming the fabric. And synchronous checkpointing that pauses training every 30 minutes shows up in throughput averages exactly like a slow interconnect; write checkpoints asynchronously to node-local NVMe and upload in the background.

What good looks like

Our reference numbers for Llama-class pretraining with FSDP2: 96-97% scaling efficiency at 4 nodes, 92-94% at 16 nodes, measured as tokens/sec versus a single-node extrapolation. If you are below 90% at 4 nodes, the money says to stop training and go find the regression — at $18.32 per node-hour, the fabric is the cheapest thing you will debug all week.

One final habit: rerun the nccl-tests baseline whenever you resize the job, change the container image, or bump NCCL versions. It takes four minutes and has caught more silent regressions for us than any other single check.