All-Reduce Traffic Estimator

Data-parallel training synchronizes gradients with an all-reduce every step. In the bandwidth-optimal ring algorithm each GPU sends and receives 2(N−1)/N times the gradient buffer, so 8 GPUs exchanging 7 billion FP16 gradients (14 GB) each move 24.5 GB per step. This all-reduce traffic estimator turns model size, GPU count and NIC speed into per-GPU traffic, total fabric traffic, step time, algorithm bandwidth and NCCL-style bus bandwidth.

☷

Job and fabric

Enter the number of GPUs in the ring, the gradient size and the per-GPU network bandwidth.

×10⁹

Usually the trainable parameter count.

Gbps

NIC line rate per GPU, one direction.

%

Protocol overhead and congestion; measure with nccl-tests.

µs

Ring all-reduce takes 2(N−1) steps.

▥

Calculation result

Per-GPU bytes = 2(N−1)/N × S. Time = 2(N−1)·α + per-GPU bytes ÷ effective bandwidth.

Data sent per GPU

—

Estimated all-reduce time

—
Gradient buffer S—
Ring factor 2(N−1)/N—
Total fabric traffic—
Bandwidth term—
Latency term—
Algorithm bandwidth S/t—
Bus bandwidth (NCCL)—

Assumes one flat ring, no compute/communication overlap and no in-network reduction. Tree, hierarchical and SHARP algorithms change the latency term and traffic pattern.

Methodology and engineering notes

METHODOLOGY

Per-GPU bytes = 2(N−1)/N × S
Time = 2(N−1)·α + per-GPU bytes ÷ effective bandwidth

How to use

  1. Enter the number of GPUs (ranks) in the ring.
  2. Enter the gradient element count, usually the trainable parameters, and the data type.
  3. Enter the per-GPU NIC bandwidth, an achievable efficiency and the per-step latency.
  4. Read per-GPU and total traffic, step time and bus bandwidth.

Ring all-reduce

A ring all-reduce is a reduce-scatter followed by an all-gather. Each takes N−1 steps in which every rank sends S/N bytes, so each rank sends 2(N−1)/N × S and receives the same amount. Total fabric traffic is N × 2(N−1)/N × S = 2(N−1) × S. This is bandwidth-optimal (Patarasuk & Yuan, 2009).

Time and bandwidth

Time ≈ 2(N−1)·α + 2(N−1)/N × S ÷ B, where B is the per-GPU bandwidth × efficiency in bytes/s. Algorithm bandwidth = S ÷ time; NCCL bus bandwidth = algorithm bandwidth × 2(N−1)/N, which is comparable to the link speed.

Worked case: 8 GPUs, 7B FP16, 400 Gbps

S = 7 × 10⁹ × 2 B = 14 GB. Ring factor 2 × 7 / 8 = 1.75, so each GPU sends 24.5 GB and the fabric carries 196 GB. At 400 Gbps (50 GB/s) and 100% efficiency the bandwidth term is 490 ms; at 80% it is 612.5 ms, plus 14 × 5 µs = 70 µs of latency. Bus bandwidth then stays just under the effective link rate of 40 GB/s.

Frequently asked questions

FAQ

How much data does ring all-reduce send per GPU?

2(N−1)/N times the buffer. For 8 GPUs that is 1.75 × S; as N grows it approaches 2 × S.

What is bus bandwidth in NCCL?

busBW = algBW × 2(N−1)/N for all-reduce. It normalizes for the number of GPUs so it can be compared with the hardware link speed.

How long does gradient all-reduce take?

Roughly 2(N−1)/N × S ÷ link bandwidth plus 2(N−1) × per-step latency. 14 GB over 8 GPUs at 50 GB/s takes about 0.49 s.

Does adding GPUs increase per-GPU traffic?

Only slightly: the factor goes from 1.5 at 4 GPUs to 1.75 at 8 and 1.98 at 128. The latency term grows linearly with N.

Is ring the only all-reduce algorithm?

No. Tree, double binary tree, hierarchical and in-network (SHARP) algorithms reduce latency at scale; ring is the bandwidth-optimal baseline used here.

Recent calculations

PROJECT RECORDS

Standards and references

ENGINEERING WORKFLOW

Continue the engineering workflow

Current workflowAI Training Fabric Planning Workflow

Estimate gradient all-reduce traffic, then size the leaf-spine fabric that carries it.

Step 1 of 2

Related tools