Designing a non-blocking fabric for 100,000 GPUs

A useful way to think about a 100,000-GPU cluster is that it is not 100,000 computers. It is one computer, and the network is most of it.

Training a large model is a loop: every GPU computes a slice of the gradient, then all of them exchange and average what they computed before taking the next step. That exchange – the all-reduce – is not overlap-and-forget background traffic. It is a barrier. The step does not advance until the slowest participant has sent and received its share, which means the cluster runs at the speed of its worst link, not its average one. A fabric that is fast on paper and lumpy in practice leaves you paying for 100,000 GPUs and training at the pace of the few thousand stuck behind a hotspot.

So when we design a fabric, the target is not peak bandwidth. It is the absence of a bad day: no oversubscribed tier, no hotspot that forms under a particular traffic pattern, no tail latency that quietly eats a fifth of your throughput. This post is how we get there, and where the genuinely hard limits are.

When the network becomes the computer

Modern accelerators live on two very different networks, and conflating them is the first mistake.

Inside a rack, the scale-up network is astonishing. A GB200 NVL72 wires 72 Blackwell GPUs into a single NVLink domain over a copper backplane, giving every GPU 1.8 TB/s to its neighbours and about 130 TB/s of aggregate bisection bandwidth. For anything that fits inside those 72 GPUs, the network is effectively free.

The scale-out network – the fabric that connects rack to rack across the hall – is a different world. The instant a collective leaves the NVLink domain and crosses into InfiniBand or Ethernet, per-GPU bandwidth falls from 1.8 TB/s to roughly 50 GB/s at 400G, and every hop adds latency. NVIDIA's own guidance puts the cost of crossing that boundary at a 5–10x latency penalty per collective. The whole art of fabric design is arranging the job so the busy, chatty traffic stays inside the fast domain and only the unavoidable minimum crosses the slow one.

Non-blocking is a promise about your worst day

"Non-blocking" has a precise meaning, and it is worth being pedantic about. A fabric is non-blocking when any half of the endpoints can talk to the other half at full line rate, simultaneously, regardless of who is talking to whom. Concretely, it means that at every switch the uplinks and downlinks are balanced 1:1 – no tier is quietly oversubscribed 3:1 to save on optics, because that 3:1 becomes your ceiling the moment the traffic pattern turns adversarial. And all-reduce traffic is adversarial by design.

Getting there takes more than buying enough switch ports. In practice a non-blocking AI fabric has to hold all of the following at once:

• 1:1 at every tier. Uplinks equal downlinks on every leaf and spine – the first corner everyone is tempted to cut.
• Full bisection bandwidth. Aggregate bandwidth between any two halves equals the bandwidth into either half. This is the property the whole design exists to protect.
• Rail alignment. Each GPU's NIC lands on its own leaf switch – its rail – so intra-rail traffic is a single hop and stays off the spine.
• Lossless transport with adaptive routing. Credit-based flow control so packets are never dropped under load, plus adaptive routing or packet spraying to steer around momentary congestion.
• In-network reduction. Summing gradients inside the switches (InfiniBand SHARP, or the equivalent on Ethernet) so the all-reduce moves far less data across the fabric.
• A separate storage and management fabric. Checkpoint and telemetry traffic never shares the compute fabric – mixing them is one of the most common and most expensive mistakes we remediate.

Miss any one of these and the headline number on the datasheet becomes fiction under real load.

Peak bandwidth is a brochure number. The figure that decides whether a training run holds is the throughput you get on your worst link, on your worst day – and that is a property of the whole fabric, not any one switch.

The tyranny of radix

Here is where the physics bites. A fat-tree – the folded Clos topology almost everyone uses – is built from switches of a fixed port count, the radix. And the radix sets a hard ceiling on how many GPUs you can wire up non-blocking before you are forced to add another tier.

The arithmetic is unforgiving:

# Napkin math for a non-blocking fat-tree.
# Rule: at every switch, uplinks must equal downlinks (1:1, no oversubscription).

def max_gpus(radix, tiers):
    """Max GPUs a k-port switch supports, fully non-blocking."""
    return radix**2 // 2 if tiers == 2 else radix**3 // 4   # 2- or 3-tier Clos

def switch_count(radix):
    """Switches in a fully populated 3-tier fat-tree."""
    return 5 * radix**2 // 4

TARGET = 100_000

for radix in (64, 128, 144):
    for tiers in (2, 3):
        capacity = max_gpus(radix, tiers)
        if capacity >= TARGET:
            print(f"{radix}-port, {tiers} tiers -> {capacity:,} GPUs "
                  f"({switch_count(radix):,} switches)")
            break

# 128-port, 3 tiers -> 524,288 GPUs (20,480 switches)
# 144-port, 3 tiers -> 746,496 GPUs (25,920 switches)
# (64-port prints nothing: it tops out at 65,536 GPUs across three tiers.)

A 64-port 400G switch – today's InfiniBand workhorse – tops out at 65,536 GPUs across three tiers. That is short of 100,000. So a 100,000-GPU non-blocking fabric is not a bigger version of a 10,000-GPU one; it forces you onto higher-radix switches, multi-plane designs, or both, and every one of those choices ripples into cabling, optics count and cost. The network is already 10 to 20 per cent of cluster capex before you buy a single extra transceiver to reach the next tier.

Rails, not just tiers

Rail-optimised topology is the lever that makes the radix go further. Instead of scattering a node's NICs arbitrarily, you put each GPU's NIC on its own dedicated leaf – its rail – so it aligns with the NVLink domain underneath. With 64-port switches, a rail-optimised design connects 32 nodes to a leaf where a naive layout manages only four: same switches, eight times the reach, and most collective traffic completing in a single hop.

The mental model we build around is simple. The NVL72 is an island where the network is free; the fabric's job is to stitch islands together while forcing as little traffic as possible to make the crossing.

Failure is a design input

None of this survives contact with reality unless you assume it is already broken. A 100,000-GPU fabric has hundreds of thousands of optical transceivers, and optics fail – they flap, they degrade, they drop packets silently. At this scale something is always down. Meta's analysis of a large training cluster found that network misconfiguration and faults accounted for more than 10 per cent of significant job failures, and that is before counting the transceivers that simply die mid-run.

So failure is not an exception to be handled after the fact; it is an input to the design. The fabric has to route around a dead link with no human in the loop, detect a degrading one before it corrupts a collective, and carry enough spare capacity that losing a switch degrades throughput rather than halting the job. A non-blocking topology that cannot survive the loss of its own links is a benchmark, not a system.

What we build

Volta's fabric is non-blocking where it has to be, rail-optimised throughout, lossless and reduction-offloaded – and it is designed as one thing with the rack, not bolted on afterwards. That last point is the one the industry keeps underpricing. A switch is a power draw and a heat load; every transceiver is watts; the fabric is a tenth to a fifth of the capex and a real slice of the megawatts. You cannot design the network in isolation from the cooling loop and the power budget, because they are the same machine – which is exactly why we own all of it, from the substation to the FLOP.

We will go deeper on the transport layer – InfiniBand versus Ethernet, in-network reduction, adaptive routing schemes – in later posts. If building fabrics like this is your idea of a good time, we are hiring.