In Mark Handley’s view, you should never bet against Ethernet. A member of OpenAI’s core networking team and a key mind behind the AI giant’s Ethernet-based Multipath Reliable Connection (MRC) protocol, Handley has pedigree in the field, with his work on Broadcom’s AI networking architecture helping lay the groundwork for the Ultra Ethernet Consortium (UEC).

“Ethernet was always likely to be a big player … with the scale of the whole industry driving forward a common, shared technology,” Handley told SDxCentral. “And as a transport protocol, we want MRC to be in the same space for its niche.” MRC is built on remote direct memory access (RDMA) over converged Ethernet (RoCE) and 800 Gb/s (800G) Ethernet interfaces.

Officially revealed by OpenAI earlier this year, the transport protocol accelerates massive-scale AI model training with greater GPU efficiency and reliability. The protocol was created with support from AMD, Broadcom, Intel, Microsoft, and Nvidia, with the shared goal to make huge fleets of GPUs behave like one tightly synchronized machine, even if failures beset the underlying network.

“What we call synchronous pre-training, which is the first stage of AI training, is the most compute-intensive piece,” Handley explained.

“When you’re doing this kind of AI training … everybody starts talking at exactly the same time, which is the worst possible workload you can think of to put onto a network, because there’s no degree of statistical multiplexing that happens here.”

RDMA and packet sprays

Handley explained that systems traditionally rely on RDMA where each communication takes one pass through the network and is pinned to a single path. But pinning each flow to a path risks exposure to collisions when pseudorandomly selected paths happen to coincide on one link and slow down operations.

“Each of them goes at half speed … if 10 of them collide, then each of them is going at a 10th of the speed,” Handley said.

MRC overcomes this slowdown by breaking down each path into many small packets, each of which takes a different path through the network. As Handley put it, when every transfer is sprayed across hundreds of paths, “you end up load balancing the network beautifully smoothly” by avoiding congestion in the core.

The MRC method doesn’t just spray packets blindly; it also tracks conditions per path, being able to avoid congested ones until the path is no longer broken.

“Now my flow is taking hundreds of paths. If one of those links fails, it’s a very small impact on that flow. And so any network failure that happens above the first top switches is basically not consequential to us,” Handley said.

Dumbing down the network

Beyond traffic patterns and topology, MRC also changes the control plane. Normally, switches run dynamic routing protocols to figure out which links are up and to route around failures. But with an endpoint-driven system that sprays across many paths and avoids failures autonomously, routers no longer need to communicate with one another. IPv6 segment routing helps endpoints select exact paths so the network adapter instructs the packet exactly where the path wants it to go.

“All the switches will do is blindly follow their orders,” Handley said. “That moves a whole set of complexity out of our networks that we don’t need anymore. Usually as you scale up, you care more and more about the network being smart in routing around failure. And actually, we did exactly the opposite, and made the network dumb. It doesn’t even try to route around failures, because MRC can route around them for us.”

Such self-healing properties fundamentally change how OpenAI coordinates with infrastructure partners such as Oracle and Microsoft, Handley said. Because MRC can detect failed or degraded paths within microseconds and immediately stop using them, network maintenance no longer requires tight synchronization with running training jobs, and as such, a lot less constant human vigilance.

“Previously, we needed to coordinate with the people running the jobs in order to say, is this a good time to reboot a switch? Now it says you can just reboot a switch. Doesn’t matter. MRC won’t care. So this gives us a whole bunch more flexibility in terms of trying to operate these really huge networks with small teams,” Handley said.

Fewer switches, more paths

Regarding switches, MRC splits an 800G interface into smaller links, allowing one interface to connect to eight different switches, creating eight parallel 100G planes instead of a single network.

Rather than linking 64 high-speed connections, it can link 512 smaller ones; with each switch reaching more GPUs, the whole cluster can be built in a less complex way. In practice, more than 130,000 GPUs can be connected using only two layers of switches instead of the traditional three or four layers, helping to reduce power use and the number of switches and optics – “there’s actually less stuff to fail, which helps,” Handley explained.

“It also gives me more GPUs connected with only one switch in the middle, which gives me a lower latency,” Handley added. “But the thing I really care about is now from the same GPU, I have eight immediate paths. And so if I lose a link that connects directly to the GPU, I still have seven paths available … We stop using that particular path, but we can recover those packets really quickly, and we’re back up to full speed very fast. This basically means that we no longer really have to worry about failures.”

Today, MRC is deployed across all of OpenAI’s largest supercomputers, including the Oracle Cloud Infrastructure (OCI)-built supercomputer and Microsoft’s Fairwater supercomputers. Infrastructure players like Nokia, Hewlett Packard Enterprise (HPE), and Arista Networks tout MRC’s benefits, as Dell’Oro Group research predicts Ethernet will generate nearly $80 billion in data center switch sales by 2030.

In the shorter term, Handley promised future generations and ongoing collaboration with OpenAI partners to “keep rolling this ball forward” amid the ongoing AI buildout.

“I’ve worked through a lot of exciting changes on the internet over the years, but the crazy scale of the things we’re trying to do right now is really difficult, and coming up with the right way of solving them is a huge challenge. It’s a lot of fun,” Handley said.

This article first appeared in SDxCentral Magazine Issue #2

To read the full issue for free, simply register.