A network switch interconnecting Clemente, Meta's GB300 NVL72 platform
– Meta

Meta detailed yet another peek behind the curtain of its AI infrastructure stack, revealing the aggregation layer stitching together its Prometheus supercluster in Ohio runs on an Arista chassis with deep-buffered Broadcom silicon.

After spilling the tea on its ice-themed custom networking stack at OCP earlier this month, the Facebook parent published yet another lengthy engineering post detailing the compute and networking stack already used to train its Muse foundation models.

Clemente, Meta's GB300 NVL72 platform
An imposing view of Clemente, Meta's GB300 NVL72 platform – Meta

While the site is planned to comprise roughly one million distributed GPUs, it currently features multiple generations of AI hardware, including an Nvidia GB200-based configuration dubbed Catalina and a GB300 NVL72 platform named Clemente.

Meta claims its Catalina platform let it deploy 800 Gb/s (800G) of scale-out bandwidth per GPU earlier than the rest of the industry. The configuration pairs each GPU with two ConnectX-7 400G network interface cards (NICs) at a 1:1 CPU-to-GPU ratio. The Blackwell Ultra-based Clemente, meanwhile, packs 72 GPUs into a rack with 20.7 terabytes of HBM3e, providing 288 GB per GPU, or 50% more than the GB200.

At the building level, each of Meta’s Ethernet-based fabric interconnects up to 20,736 GPUs through the company’s latest custom network fabric, called non-scheduled fabric or NSF. Following its previous Disaggregated Scheduled Fabric, NSF uses shallow-buffer switches capable of handling 51.2 terabits of bandwidth. Both are based on the hyperscaler’s own network operating system, Facebook Open Switching System (FBOSS), allowing on-site engineers to rapidiy iterate and optimize the fabrics on the fly.

Meta also provided an even greater look at its centralized, Ethernet-based “super-spine” for interconnecting multiple disparate fabrics – backend aggregation (BAG), which was unveiled during OCP.

In its latest engineering post, Meta revealed that for its Prometheus site, the multi-layer hub-and-spoke network is built on Arista’s 7800-series modular switch chassis, which house Broadcom Jericho3 silicon.

Across Prometheus, the network spans millions of miles of fiber and tens of thousands of switches, Meta uses short-reach, low-cost pluggables inside the buildings while for links beyond roughly 10 km (6 miles), 800G-ZR coherent optics keep BAG connections flowing.

Meta is far from finished with the New Albany-based site, with plans to supplement its compute platforms with Vera Rubin NVL72s via its upcoming San Miguel platform, integrating directly into Prometheus’s existing network fabric

Lessons learned from Prometheus will also be applied to its upcoming supercluster in Richland Parish, Louisiana. Previously a two-gigawatt development, the site, christened Hyperion, has jumped to 5GW, initially encompassed nine buildings, each coming online in phases through 2030.