Meta has provided a detailed glimpse into its next-generation AI infrastructure, with multi-terabit networking and open systems set to power workloads for its Muse agents.

At OCP in San Jose, Nvidia already revealed to the world Meta’s custom scale-out network switch, called IceCubes, built on Spectrum-6. But the hyperscaler is hedging its bets when it comes to switch ASICs, with it revealing the switches would also use Broadcom’s Tomahawk 6.

Both sets of chips provide 102.4 terabits per second (Tb/s) of bandwidth – a two-fold increase on previous platforms. They’ll run Meta’s open-source Facebook Open Switching System (FBOSS) network operating system to dynamically route packets across a multiplane configuration, distributing GPU traffic across up to eight independent network planes, providing for fewer layers of switches and therefore improved resiliency should a system fail.

Beyond opting for diversity in switching hardware, Meta also provided more details on the network racks set to house them.

IcePacks are based on the OCP’s ORV3-HPRv1 standard and are the hyperscaler’s first-ever liquid-cooled network racks. Each blade inside supports 128 separate 800G ports operating simultaneously, and due to its non-blocking design, every single port can run at maximum capacity without causing bottlenecks under heavy AI loads, ensuring GPU utilization.

IceCube scale-out switches were joined by a less chilly-sounding scale-up switch: Tahan.

Built on a switch ASIC from just one vendor, Broadcom’s Tomahawk 5, each provides 51.2 Tb/s of total bandwidth. Unlike its larger scale-out cousin, the Tahan switch is highly condensed by design, with four switches interconnected to 16 of Meta’s MTIA 300 custom silicon chips to provide 6.4 Tb/s per accelerator.

The switch also reuses the same remote direct memory access (RDMA) over converged Ethernet (RoCE) transport as Meta’s scale-out network to form a unified fabric across both scale-up and scale-out that’s united via FBOSS, keeping the entire fabric neatly tied together.

Meta also provided a peek at its next-gen scale-up blade. No name affixed, but it’s also set to be built on Broadcom’s Tomahawk 6 chips – eight of which can be interconnected to 72 accelerators per rack, marking a sizable increase over Tahan. Besides boosting density, the 102.4T gen blade will also support Ethernet for Scale-Up Networking (ESUN). No timeframe for when this next-gen scale-up blade will debut was given.

Fabric multiplane makeover & scale-across aggregation

Beyond custom hardware designs, Meta’s OCP news saw its engineers reveal a new multiplane backend scale-out fabric.

Dubbed Flattened Planar Fabric (FPF), it replaces traditional single-plane uplinks by connecting each GPU through eight independent 100G links to eight distinct switches, one per network plane. The idea is to improve fault tolerance by allowing the host to steer traffic away from downed planes. The switch from a three-tier to a two-tier fabric, meanwhile, removes the need for more switches, which in turn reduces unnecessary hops and lowers both latency and power.

Meta revealed Nvidia engineers worked closely with the Facebook parent on developing FPF, tapping innovations from Spectrum-X multiplane configurations. The pair also worked with unnamed cloud partners to “help standardize multiplane fabric implementations” based on its learnings with FPF.

“This work with cloud providers on FPF is just one example of how we at Meta are not just building our own infrastructure, but are helping our partners’ infrastructure,” the post, authored by members of Meta’s network infrastructure team, reads.

On top of scale-out fabric rework, Meta’s OCP unveilings also saw it detail backend aggregation (BAG), a centralized, Ethernet-based “super-spine” for interconnecting multiple disparate fabrics into one congealed mega cluster.

The scale-across concept works for systems spanning buildings, campuses, and even facilities spread across regions, with Meta claiming it allows for capacity between sites to reach tens of petabits per second.

BAG’s development marks Meta's development of vast gigawatt-scale facilities, including Prometheus in Ohio and Hyperion in Louisiana. The backend aggregation layer is designed to allow these distributed sites to connect as a single cluster. It uses what Meta network engineers billed as “eBGP link-bandwidth signaling,” which sees traffic distributed in proportion to the capacity actually available on each path, with damping in place to prevent a single link failure resulting in routing churn across the stack.

BAG also encompasses adaptive routing improvements to reduce latency and scheduling aimed at minimizing how much collective traffic needs to cross the layer.

“Training remains resilient throughout. When a replica fails, it is automatically excluded, and the surviving replicas continue synchronizing across BAG,” Meta engineers wrote. “The direction of travel is longer distances still – training jobs spanning regions tens, hundreds, or thousands of kilometers apart – and increasingly heterogeneous clusters that mix hardware generations and vendors.”