OpenAI unveiled specifications for large-scale AI training networks developed alongside an all-star lineup.

AMD, Broadcom, Intel, Microsoft, and Nvidia lent expertise to create Multipath Reliable Connection (MRC), a solution said to speed up AI model training on supercomputers with greater graphic processing unit (GPU) efficiency and reliability.

The tool’s unveiling offers a rare under-the-hood peek from the firm, with OpenAI keen to make the approach available for others to implement and build on. MRC, it explained, is a response to large AI-training jobs being sensitive to network delay, congestion, and failures. For example, a single late transfer can stall GPUs, and at large scale even strong networks, like those of OpenAI that boast millions of links, see link and device failures.

OpenAI pointed to synchronous pretraining, the initial large-scale training phase where one AI model is trained across many GPUs and servers at the same time, which all have to stay in sync at every training step. Such pretraining is especially vulnerable because many GPUs cooperate in lockstep, so any single link issue or failure can ripple through the whole job and act like a “failure amplifier.”

MRC, which is built into the latest 800 Gb/s (800G) network interfaces, lets one transfer be spread across hundreds of paths by extending remote direct memory access (RDMA) over converged Ethernet (RoCE). The solution draws on techniques developed by the Ultra Ethernet Consortium (UEC) and extends them with IPv6 segment routing (SRv6)-based source routing. Such routing lets the sender choose the path the packet should take through the network, meaning a GPU network interface can explicitly pick one of many available paths to support large-scale AI networking fabrics.

OpenAI also splits an 800G interface into smaller links, meaning one interface can connect to eight different switches, creating eight parallel 100G planes rather than a single network. Instead of linking 64 high-speed connections, it can link 512 smaller ones; with each switch reaching more GPUs, the whole supercomputer can be built in a simpler way. OpenAI claimed more than 130,000 GPUs can be connected using only two layers of switches instead of the traditional three or four layers, helping to reduce power use, component count, and cost while increasing path diversity and keeping more traffic local to immediate tier-zero switches, which can improve performance.

“This gives us enough redundancy to ride out network failures, while using less power than equivalent three- or four-tier single-plane networks,” OpenAI noted. “Second, MRC’s adaptive packet spraying load-balances well enough that we see essentially no congestion in the core of the network. This greatly reduces variation in throughput between flows during synchronous training, where eliminating outliers is central to performance. It also means that when multiple jobs share the cluster, they do not impact one another’s performance.”

MRC is currently deployed across all of OpenAI’s largest supercomputers, including the Oracle Cloud Infrastructure (OCI)-built supercomputer in Abilene, Texas, and Microsoft’s Fairwater supercomputers. The AI giant said it has already used MRC to train multiple of its frontier models, leveraging hardware from Nvidia and Broadcom.

With UEC as its basis, MRC follows various moves pushing Ethernet further into the network space in response to AI-driven demands.

The Open Compute Project (OCP), for example, unveiled the Ethernet for Scale-Up Networking (ESUN) working group at last year’s OCP Global Summit. ESUN has progressed through work by Microsoft and Meta, with the firms collaborating on a specification for tackling scale-up GPU connectivity and other AI-heavy deployments.