Artificial intelligence (AI) and high performance computing (HPC) have pushed requirements to the limit for compute, storage capacity and networking resources. More powerful chips have been a primary factor in driving rack densities higher and higher. A decade ago, the average processor consumed less than 100 watts of power. Today, they consume around 500 watts.

The more power that becomes available in a smaller space, the more application developers come up with ways to harness it – and then they demand even more compute power. Earl Joseph, CEO of Hyperion Research, said HPC growth is largely surging due to the advent of more demanding applications.

“AI, machine learning and deep learning are growing at close to 30% a year,” Joseph said.

Let’s look at how density has evolved and how high it might go.

Data centers are being planned, ordered and commissioned in record numbers. This is shown in statistics such as a 50% increase being forecast for the power footprint of data centers by 2025, as well as 500% growth in global data generated between 2019 and 2025, according to the Uptime Institute. Data centers are being built in unprecedented numbers and at far higher compute densities than ever before. Data centers need to pack more punch per rack or per square foot to keep up with demand.

Uptime Institute numbers from just over a decade ago put the average power densities per rack in the 4–5 kW range. By 2020, that was up to 8–10 kW per rack. Note, though, that two-thirds of U.S. data centers surveyed said that they were already experiencing peak demands in the 16–20 kW per rack range. The latest numbers from 2022 show 10% of data centers reporting rack densities of 20–29 kW per rack, 7% at 30–39 kW per rack, 3% at 40–49 kW per rack and 5% at 50 kW or greater. Clearly, modern applications and data volumes are pushing rack density to unprecedented heights.

Hyperscale data centers from the likes of Amazon, Facebook, Google and Microsoft have been among the leaders in density growth over the past decade. They have pioneered ways to better cool and power data centers while providing as much compute power as possible in a tiny footprint. Some have evolved racks of 50 kW and even higher.

What you tend to find, however, is that the hyperscalers are now leaving the highest densities to others. They took the industry up to new levels of density but now prefer to flatten out at densities around the 30 kW range and slightly below. As they operate at massive scale and high efficiency, they can satisfy most user demands. What the hyperscalers want is optimization. They want a high (but not too high) rack density that is scalable and available at an attractive price point.

For the HPC and AI markets, what is emerging are specialist providers that cater to the highest densities. These companies are gaining business from unexpected sources.

“While HPC used to be the province of businesses and research entities with a billion-dollar turnover, much smaller businesses are now using it for competitive advantage,” said Mike Andrea, CEO of Oper8 Global.

He noted the democratization of HPC applications from large, well-funded organizations down into smaller-scale research and development outfits, as well as into fields such as aerospace, seismic engineering, 3D modeling, autonomous vehicle simulation, various AI use cases, energy, oil and gas production, weather forecasting, analytics, healthcare, and 3D film rendering. The list of those making heavy usage of HPC continues to grow.

“Latency remains a primary driver in HPC, along with the ability of data centers to support rack densities in excess of 100 kW per rack,” said Andrea, whose company is working with several customers that are demanding anywhere from 80 kW to 200 kW per rack.

To compete in that market, highly dense data centers must be situated near to their customers to eliminate latency. Thus, these HPC specialists are likely to be found only in certain regions where there are clusters of customers making outrageous density demands.

Another trend is for data centers to cater to a small subset of the growing HPC market by deploying only a rack or two of high-density gear. Some edge and colocation data centers are beginning to implement one extreme-density HPC pod of two to 12 racks, or a cluster of several HPC racks alongside racks of more modest densities. Such a strategy helps data centers satisfy the requirements of one or two clients and build up their HPC arsenal without having to massively invest in a complete redesign of the entire data center.

That said, there is plenty of work involved in bringing in even a rack or two. As well as new servers and supporting gear, cabling and other improvements will be needed. It also presumes that sufficient additional power is on hand and that HPC racks can be served by enough cooling. Hence, high-density data centers will probably need to invest in computational fluid dynamics (CFD) technology to enhance air and cooling flows to avoid hot spots.

Further, they require advanced cooling technologies and perhaps even liquid cooling of some kind to keep temperatures down in their new racks. Very hot racks can lead to failures in power distribution units (PDUs) due to the presence of so much rear-rack hot air. This is especially the case with racks more than 35 kW.

“Where high-power consumption elements are packed into limited space, liquid cooling may become a necessity,” said Onur Celebioglu, senior director of engineering, HPC, and emerging workloads at Dell Technologies.

Water-based cooling options such as actively cooled rack doors, direct liquid cooling with cold plates, and immersion cooling are becoming commonplace in HPC racks. But be warned. Liquid-cooled HPC applications may necessitate wider and deeper cabinets to accommodate extra power feeds and fluid manifolds. The cost, coupled with the need for serious reconfiguration of an existing data center, may cause some to avoid the HPC market.

Not so long ago, highly dense racks were thought be 10 kW or more. That looks positively miniscule in light of modern density numbers. No one knows how high density can go. But get ready for some outrageous figures in the coming years.

“Today, high-density racks are roughly 40 kW to 125 kW and extreme-density racks are up to 200 kW and even beyond,” said Andrea.