SAN JOSE, Calif. — Facebook needed to “reimagine” its data centers to accommodate growing demand and limited power and space, executives said at the Open Compute Project (OCP) Global Summit today. The end result is two new modular switches: Minipack and a block switch jointly developed with Arista.
The companies contributed both Minipack and the Arista 7368X4 switch to OCP. The Arista 7368X4 runs Arista’s EOS operating system, Facebook’s FBOSS operating system, “or any other operating system you like,” said Arista Networks COO Anshul Sadana.
It has a four-rack unit design and all active components are removable. Compared to a traditional modular platform, it doubles system density with a 60 percent reduction in power at under 10 watts per 100 G port, the companies say.
This high-density, low-power platform was increasingly important to Facebook as it looked to accommodate massive demand — more than 2.6 billion people now use its services — within the physical constraints of power and real estate coupled with optics supply availability.
“One of the things we had to do over the past several years is rethink the topologies, the hardware, and the software,” said Omar Baldonado, director of network engineering at Facebook. “We could see that change had to happen inside the data center, so the teams came together to rethink network hardware and software, and these are the components.”
Data Center Fabric
The first piece is F16, Facebook’s next-generation data center fabric design. It has four-times the capacity of the earlier design, and it’s more scalable and simpler to operate, Baldonado said.
F16 is based on the Broadcom Tomahawk 3 ASIC, but instead of four multi-chip based planes with 400G link speeds, it uses the ASIC to create 16 single-chip based planes with 100G link speeds — also important from a lower-power-use perspective.
“It is simpler and it is flatter,” compared to the earlier fabric, Baldonado said. “It has far fewer hops, which means better performance. It’s simpler to manage with fewer control planes, and it uses less power.”
Facebook also upgraded its Fabric Aggregator, which was announced last year. Fabric Aggregator is a disaggregated design for connecting data centers within a region. The new version, called HGRID, uses all the same design principles. But now the building block is the Minipack platform, which powers the F16 fabrics.
Minipack uses half the power and space compared to its predecessor, Backpack, Baldonado said. He added that “it also has a modular architecture, which makes it more flexible.”
The new switch is built by Edgecore and it runs Facebook’s FBOSS software on top. The companies also partnered with Cumulus Networks to make available a commercial network operating system — Cumulis Linux — to support Minipack. The new platform can run the SONiC/SAI open source stack originally developed by Microsoft.
Facebook also custom designed a microserver called MiniLake for the control plane of Minipack. And it contributed the full design package of Minipack and MiniLake to OCP. This includes the system hardware specification, the electrical design files, the mechanical design files, and programming images.