The Open Compute Project (OCP) released a new specification for Ethernet-based scale-up networking, as driven by collaboration between Meta and Microsoft.

The Ethernet for Scale-Up Networking (ESUN) 1.0 specification aims to tackle scale-up GPU connectivity and other AI-heavy deployments, with ESUN presented as an add-on to existing services instead of replacing them. This approach, the team behind it argues, preserves backward compatibility while protecting prior hardware and software investments.

The whitepaper was co-authored by Manoj Wadekar, AI systems technologist at Meta, and Rajesh Sankaran, distinguished engineer and vice president for Azure hardware systems and architecture at Microsoft. The pair argued that costly custom solutions and streamlined deployment can be reduced by defining ESUN header requirements, congestion management, and differentiated traffic handling.

Specifically, this means replacing the typical IP/UDP header stack, usually 28-48 bytes, with a compact 4‑byte ESUN header, aiding layer-two encapsulation – the data layer link in an Ethernet frame – to handle scale-up traffic.

Under the proposed header, a special EtherType field – the ESUN Header EtherType, with its numerical value still pending assignment – will mark frames so that switches and hosts can recognize them as carrying such traffic.

“Ethernet Scale-Up Network provides higher goodput efficiency for all the message sizes with this change,” the authors explained. “Optimized packet headers reduce protocol overhead, improving goodput and lowering latency, which maximizes compute utilization and reduces energy consumption. Enhanced congestion management and load balancing further boost operational efficiency, minimizing idle time for expensive GPU resources.”

With modern AI platforms demanding more efficient, resilient, and scalable interconnects to support large GPU domains, the ESUN spec aims to standardize Ethernet enhancements, while complying with the OCP tenet that contributions are capable of serving Fortune 100 and large hyperscale clients.

OCP ESUN also proposes use of link level retry (LLR) to achieve link resiliency against link level errors, a featured defined by Ultra Ethernet Consortium (UEC) to make high-speed links resilient against packet drops due to high bit error rates (BER).

The Meta and Microsoft engineers also recommended an additional link-level congestion control mechanism for credit-based flow control (CBFC), also defined by UEC, which helps provide more memory efficient designs for ESUN switch devices.

The specification adds security considerations in light of the ESUN header not being compatible with IPSec through its replacement of the IP header. In its place, MACSec can be used for protection at the layer-two level, providing point-to-point link-level encryption. End-to-end encryption, meanwhile, operates at higher levels in the stack, outside the scope of ESUN.

OCP’s focus on AI scale-up comes amid increased demand for AI services, leading to the project setting up the ESUN working group last October, with Meta and Microsoft joined by vendors like Broadcom, Cisco, Hewlett Packard Enterprise Networking (formerly Juniper Networks), Nvidia, and OpenAI.

The group has grown to 175 companies since the launch, further bolstering Ethernet's dominion in the AI data center era.

While InfiniBand held an 80% share of the AI back-end networks market as of 2023, recent Dell’Oro Group research suggests its dominance is waning, with Ethernet-based offerings projected to drive nearly $80 billion in data center switch sales over the next five years.

This is down to Ethernet being well-known to the vast majority of industry engineers compared to proprietary alternatives, as well as making it considerably cheaper to interconnect hundreds-of-thousands of GPUs.