All-Reduce Traffic Estimator
Data-parallel training synchronizes gradients with an all-reduce every step. In the bandwidth-optimal ring algorithm each GPU sends and receives 2(N−1)/N times the gradient buffer, so 8 GPUs exchanging 7 billion FP16 gradients (14 GB) each move 24.5 GB per step. This all-reduce traffic estimator turns model size, GPU count and NIC speed into per-GPU traffic, total fabric traffic, step time, algorithm bandwidth and NCCL-style bus bandwidth.
Job and fabric
Enter the number of GPUs in the ring, the gradient size and the per-GPU network bandwidth.
Usually the trainable parameter count.
NIC line rate per GPU, one direction.
Protocol overhead and congestion; measure with nccl-tests.
Ring all-reduce takes 2(N−1) steps.
Calculation result
Per-GPU bytes = 2(N−1)/N × S. Time = 2(N−1)·α + per-GPU bytes ÷ effective bandwidth.
Data sent per GPU
—Estimated all-reduce time
—Assumes one flat ring, no compute/communication overlap and no in-network reduction. Tree, hierarchical and SHARP algorithms change the latency term and traffic pattern.
Methodology and engineering notes
METHODOLOGY
Time = 2(N−1)·α + per-GPU bytes ÷ effective bandwidth
How to use
- Enter the number of GPUs (ranks) in the ring.
- Enter the gradient element count, usually the trainable parameters, and the data type.
- Enter the per-GPU NIC bandwidth, an achievable efficiency and the per-step latency.
- Read per-GPU and total traffic, step time and bus bandwidth.
Ring all-reduce
A ring all-reduce is a reduce-scatter followed by an all-gather. Each takes N−1 steps in which every rank sends S/N bytes, so each rank sends 2(N−1)/N × S and receives the same amount. Total fabric traffic is N × 2(N−1)/N × S = 2(N−1) × S. This is bandwidth-optimal (Patarasuk & Yuan, 2009).
Time and bandwidth
Time ≈ 2(N−1)·α + 2(N−1)/N × S ÷ B, where B is the per-GPU bandwidth × efficiency in bytes/s. Algorithm bandwidth = S ÷ time; NCCL bus bandwidth = algorithm bandwidth × 2(N−1)/N, which is comparable to the link speed.
Worked case: 8 GPUs, 7B FP16, 400 Gbps
S = 7 × 10⁹ × 2 B = 14 GB. Ring factor 2 × 7 / 8 = 1.75, so each GPU sends 24.5 GB and the fabric carries 196 GB. At 400 Gbps (50 GB/s) and 100% efficiency the bandwidth term is 490 ms; at 80% it is 612.5 ms, plus 14 × 5 µs = 70 µs of latency. Bus bandwidth then stays just under the effective link rate of 40 GB/s.
Frequently asked questions
FAQ
How much data does ring all-reduce send per GPU?
2(N−1)/N times the buffer. For 8 GPUs that is 1.75 × S; as N grows it approaches 2 × S.
What is bus bandwidth in NCCL?
busBW = algBW × 2(N−1)/N for all-reduce. It normalizes for the number of GPUs so it can be compared with the hardware link speed.
How long does gradient all-reduce take?
Roughly 2(N−1)/N × S ÷ link bandwidth plus 2(N−1) × per-step latency. 14 GB over 8 GPUs at 50 GB/s takes about 0.49 s.
Does adding GPUs increase per-GPU traffic?
Only slightly: the factor goes from 1.5 at 4 GPUs to 1.75 at 8 and 1.98 at 128. The latency term grows linearly with N.
Is ring the only all-reduce algorithm?
No. Tree, double binary tree, hierarchical and in-network (SHARP) algorithms reduce latency at scale; ring is the bandwidth-optimal baseline used here.
Recent calculations
PROJECT RECORDS
Standards and references
ENGINEERING REFERENCES
- NVIDIA nccl-tests PERFORMANCE.md – “Bus bandwidth”: busBW = algBW × 2(n−1)/n for all-reduce
- Patarasuk & Yuan, J. Parallel Distrib. Comput. 69(2), 2009: bandwidth optimal all-reduce algorithms for clusters of workstations
- Thakur, Rabenseifner & Gropp, IJHPCA 19(1), 2005: optimization of collective communication operations in MPICH
The efficiency and per-step latency are user assumptions; calibrate them with nccl-tests on the target fabric before capacity decisions. Last reviewed: 2026-10-20.
ENGINEERING WORKFLOW
Continue the engineering workflow
Estimate gradient all-reduce traffic, then size the leaf-spine fabric that carries it.