While virtualization and the cloud have dramatically improved server utilization, they have also turned servers and server clusters into amorphous “black boxes” that have made it difficult, if not impossible, to optimize software for peak performance.

One solution to this problem is to scale data center resources up and/or out as needed to achieve the desired performance. But such brute force scalability is costly, and creates unnecessarily high demands on data center space and power.

A more scalable and cost-effective solution is possible by taking a component-level approach to server clusters in an integrated design that is capable of delivering continued improvements in cluster-wide performance, much like Moore’s Law has done for integrated circuits.

This article explores the underlying causes of performance bottlenecks in server clusters today and introduces the component-level architecture as a possible solution.

The problem is a familiar one to DevOps teams and software engineers: An application that appears to operate perfectly well during development begins to experience performance problems—often crippling ones—when running at “full speed” in production.

The usual cause of the problem is rooted in the way server clusters are built. Racks get populated with hyperconverged servers containing the compute and storage resources, with a top-of-rack (ToR) network switch providing the east-west and north-south communications.

Such “rack-and-stack” configurations are often engineered with two bottlenecks. One is in the network where depending on how the virtualized and/or containerized workloads get assigned, an application’s peak bandwidth demand can exceed the capacity of either the server’s virtual network interface or the ToR switch—or both.

The other bottleneck involves the way storage is often configured. For example, SAS expanders are a cost-effective way to add shared or dedicated hard disk drives and solid state disks (SSDs) to a cluster of servers. Some expanders are capable of supporting over 128 drives, each of which can support a throughput of up to 6 Gb/s. These expanders typically connect to servers equipped with a small number of SAS interfaces that top out at 12 Gb/s each, which means the configuration is almost certain to be over-subscribed, perhaps substantially. Even hyperconverged systems with lots of direct-attached storage can be (and often are) over-subscribed.

Scaling these configurations up and out can help, but doing so often only worsens or simply shifts any performance bottlenecks by placing more I/O demand on a ToR switch or storage subsystem that is already over-subscribed and/or experiencing high latency. One way to eliminate these bottlenecks is to take a component-level approach to building server clusters, and that is the rationale behind the “cluster-in-a-box.”

Engineering a rack-and-stack server cluster for optimal resource utilization and high performance is difficult, but it can be done. The problem is attempting to perform this difficult engineering task one rack at a time. A better approach would be to match all of resources in the cluster—compute, storage, and networking—to fully utilize each in a non-blocking manner and, therefore, achieve peak performance and price.

Such component-level “ultra-convergence” is the approach being taken by a new category of systems: the cluster-in-a-box. For distributed and edge computing applications, where the cluster-in-a-box provides a total solution, the system could be considered a complete “cloud-in-a-box.” For large-scale needs, multiple cluster-in-a-box systems could be incorporated into larger clusters via the data center’s leaf and spine network.

In these and other cloud infrastructures, the cluster-in-a-box must be able fulfill some fundamental needs. The obvious one is eliminating the latency and throughput bottlenecks that prevent rack-and-stack configurations from delivering peak performance. And the best way to assure that is with a full non-blocking architecture for both storage and networking I/O. For example, each SSD would have its own, dedicated serial advanced technology attachment (SATA) or non-volatile memory express (NVMe) interface, and the integral switch would be able to support the peak bandwidth demand for each and every embedded server.

Another obvious need is for a “bring your own software” (BYOS) openness that enables organizations to choose whatever bare metal, virtualized and/or containerized environment and management software that best meets the requirements of their various applications. And this can be accomplished by utilizing standard merchant silicon and commercial off-the-shelf (COTS) components needed to assure software compatibility and interoperability.

High availability is always a fundamental need in both public and private clouds, and the ways to achieve that are now well-established best practices. Every cluster-in-a-box system should, therefore, be expected to have dual power supplies and dual active switches, and these along with the individual server modules should all be hot-swappable. Support for BYOS makes it possible to implement application-level high availability in software both within and across cluster-in-a-box systems.

Energy efficiency has become an increasingly important need both at the edge, where providing power can be expensive, and in the core, where available power is at risk of being exhausted by the relentless growth in infrastructure. By matching compute, storage, and networking resources for optimal utilization, the cluster-in-a-box wastes nothing. And being fully self-contained also eliminates the power and cooling inefficiencies inherent in rack-and-stack configurations built using disparate systems.

Perhaps the most important need fulfilled by the cluster-in-a-box is the one for continued gains in price/performance at the system level, much like Moore’s Law has done for integrated circuits (ICs). And just as for ICs, the key here is density. Cramming more transistors into every successive generation of CPUs, for example, has made it possible for today’s smartphones to have more processing power than many data centers had just two decades ago.

So while the business case for the cluster-in-a-box is already quite compelling, it can be expected to get even stronger over time. Within the same compact chassis will come more and faster CPU cores and memory, more storage, and faster switching—all matched for high utilization and together delivering higher performance at little or no additional cost.

For all these reasons, these boxes without bottlenecks can be expected to usher in a new generation of server clusters and clouds from the edge to the core.