Interconnect: NVLink vs InfiniBand

An interconnect is the high-speed communication link that transfers data between GPUs within a server (intra-node, e.g., NVLink) or between servers in a cluster (inter-node, e.g., InfiniBand); its bandwidth and latency are primary bottlenecks in distributed AI training and inference.

An interconnect is the high-speed communication link that transfers data between GPUs within a server (intra-node, e.g., NVLink) or between servers in a cluster (inter-node, e.g., InfiniBand) — in the NVLink vs InfiniBand comparison, intra-node links are typically 5–50× faster, and interconnect bandwidth and latency are primary bottlenecks in distributed AI training and inference.

NVLink, introduced in 2016, has evolved through several generations. NVLink 4.0 in H100 systems provides 900 GB/s of bidirectional bandwidth per GPU when all eight GPUs in a node communicate through NVSwitch simultaneously — compared to approximately 128 GB/s for PCIe 5.0. InfiniBand, originally developed for high-performance computing (HPC), offers per-port bandwidths of 400 Gb/s (NDR, 2022) and 800 Gb/s (XDR, 2025) with sub-microsecond latency, enabling the all-reduce collective operations that synchronize gradients across thousands of nodes. Google's TPU pods use a proprietary inter-chip interconnect (ICI) with optical circuit switching between pods for their own training infrastructure.

Interconnect performance determines how efficiently a cluster scales with added nodes. If inter-node bandwidth is far below intra-node bandwidth, parallelism strategies must minimize cross-node data movement: tensor-parallel groups (which exchange partial sums on every layer) are confined to one node's NVLink domain, while pipeline parallelism (which only passes activations between stages) spans nodes over InfiniBand. All-reduce bandwidth — the collective that sums gradients across all participating GPUs — directly sets an upper bound on training throughput; if all-reduce time exceeds backward-pass time, GPUs idle.

As of 2026, NVIDIA's 800G InfiniBand and the NVLink Switch domain — capable of connecting up to 576 GPUs at full NVLink speeds across multiple chassis — represent the frontier. Competing approaches include Broadcom's Ultra Ethernet standards and silicon photonics for inter-rack optical links that reduce power consumption. The network topology of a cluster (fat-tree, rail-optimized, or dragonfly) is a primary differentiator between well-engineered and poorly-engineered AI infrastructure, as important to total training throughput as raw GPU FLOPS.