Meta is building its own network switch to run multiplane networking for its next-generation AI clusters based on Nvidia’s Spectrum-6 switching tech.
Meta’s own Spectrum-6-infused switch will be placed in liquid-cooled network racks called IcePacks. They’ll run Meta’s Facebook Open Switching System (FBOSS) network operating system to route packets across its so-called AI factory infrastructure, which encompasses wider Nvidia Vera Rubin offerings.
Nvidia helped develop the switch abstraction interface and Open Compute Project (OCP) standard around FBOSS, with the OS already enjoying support from earlier Spectrum Ethernet switches.
Central to the effort is that Meta’s switches can now support multiplane configurations, distributing GPU traffic across up to eight independent network planes. The pair contended that the concept allows the hyperscaler to scale to 100,000+ GPUs with fewer layers of switches, while allowing for improved resiliency should a component fail.
“We are excited about our close partnership with Nvidia in making multiplane fabrics operationally scalable and performant, a critical requirement both for our own infrastructure builds and for our joint cloud partners,” said Omar Baldonado, Meta’s senior director for data center and AI networking.
“Extending FBOSS to Spectrum ASICs means we can run that same software stack across our fabrics, and Meta’s new Spectrum-based switch, IceCube, brings that into IcePack – our liquid-cooled 102.4T network rack for scale-out that connects upcoming generations of accelerators in our fleet.”
Spectrum-6 switches formally debuted in July, with Meta joining the likes of Microsoft, CoreWeave, Nebius, and SpaceXAI among the initial tranche of customers lining up for access.
Unlike its contemporaries, however, Meta’s use of the underlying ASIC marks its latest DIY approach to its infra, with the firm tending to open-source large portions of its stack. FBOSS, for example, is open-source, though Nvidia execs said they were unable to confirm what plans Meta had for the FBOSS switch.
Comments