Hefting a massive 50 pound GPU from his oven, Nvidia CEO Jensen Huang unveiled the company's first AI accelerator based on the all-new Ampere architecture during Thursday's virtual Nvidia GTC event.

According to Huang, the A100 GPU provides data centers with an artificial intelligence (AI) inference and training accelerator that doesn't compromise between high-performance and flexibility.

"The type of accelerators we've offered to data centers over the last several years have been optimized for different tasks," he explained. "Yet the data centers really want to be high performance as well as flexible at the same time."

The chip itself is built on Taiwan Semiconductor Manufacturing Company's (TSMC) 7-nanometer manufacturing process based on Nvidia's third-generation Tensor Core architecture using a chip-on-wafer-on-substrate packaging technology. This technology, Huang explains, puts the memory on the same substrate as the processor itself. Combined with a massive bank of second-generation high-bandwidth memory (HBM2), the A100 is capable of 1.6 TB/s of frame-buffer bandwidth.

According to Nvidia, The A100 boasts 250% higher performance in 16-bit floating-point calculations compared to the previous generation Volta-based V100 GPUs, and nearly 500% higher performance when taking advantage of sparsity. That performance advantage balloons in 32-bit floating-point equations where the A100 reportedly offers a 1,000% advantage over the older chip.

This is possible thanks to the GPU's third-generation Tensor Core architecture which features a new numerical format for 32-bit floating-point equations called Tensor Float 32. Nvidia claims this format offers the range of 32-bit equations at the precision of 16-bit.

"You input in FP32, it processes it with Tensor Float 32, and accumulates it in FP32," Huang said. "As a result, no code change is necessary when you train."

According to Karl Freund, senior analyst of machine learning and high-performance computing (HPC) at Moor Insights and Strategy, Nvidia's announcement broke the mold of what we've come to expect in terms of performance.

"Like the V100 from 2017, the A100 appears to surpass even the most optimistic expectations of many industry observers," he wrote in an article contributed to Forbes.

Introducing the MIG

Addressing the need to scale out AI workloads, Nvidia developed what it's calling Multi-Instance GPU (MIG), which enables a single A100 chip to be segmented into multiple smaller GPUs.

"Its the ability to turn one GPU into many," Huang explained. "You can have one GPU, you can have up to seven independent GPUs or some combination in between."

This addresses one of the challenges faced by cloud providers when offering GPU acceleration by allowing multiple customers to access a single GPU at any given time.

"For inference or public cloud, instead of having one person use a GPU, fractionalize it, create seven different instances so that each one of the customers can rent a smaller computer," Huang said. "You now have the ability to have one data center architecture for acceleration that is flexible, high throughput, and enables higher utilization."

Big Performance Itty Bitty Form Factor

The Ampere-based A100 GPU will be available in a variety of form factors including a data center focused AI and machine learning (ML) offering called the DGX A100, the motherboard for which Huang had "baking" in his oven leading up to the announcement. The DGX A100 is actually Nvidia's third-generation AI offering.

"It's fully optimized and you simply take it out of the box, plug it in, and you have a state-of-the-art development system for AI," he said.

However, unlike previous generations of the DGX line that were optimized for training, this is Nvidia's first to unify training and inference workloads. And thanks to MIG, Huang said it is now possible to run up to 56 separate workloads on a single DGX board, with "each one of them could have the equivalent performance of a Volta."

At the heart of the machine are eight A100 GPUs, two 64-core AMD Rome CPUs, 1 TB of system memory, nine Mellanox ConnectX-6 virtual protocol interconnect (VPI) NICs, six Nvidia NVSwitches for simultaneous communications across each of the GPUs, and 15 TBs of NVMe-based storage.

"This is the first computer ever made that in one node exceeds five petaflops of computing capability," Huang boasted.

However, it should be noted that performance is achieved using 16-bit floating-point calculations and the new system's performance is actually half that for 32-bit mathematics. This performance, however, has another benefit in offering customers significantly reduced cost of ownership and far lower power consumption.

"$11 million, 630 kilowatts is approximately the going price for a state-of-the-art AI data center," Huang said.

Using five DGX A100s running in a single rack, Nvidia claims to be able to provide similar performance for approximately $1 million and just 28 kilowatts. These saving are crucial as sustainability and the carbon footprint of IT infrastructure is becoming increasingly important for data centers and cloud providers.

Additionally, as part of Nvidia's membership in the Open Compute Project (OCP), the company will be making A100 available to hyperscale infrastructure vendors.

Inspur is one of the first to announce products based on Nvidia's new data center graphics chips. The vendor plans to pack as many as 16 A100 GPUs into the larger hyperscale chassis.

In fact, Nvidia boasts early commitments from Google, AWS, Microsoft, Alibaba, Dell, Lenovo, and Hewlett Packard Enterprise (HPE), to name just a few. This, according to Freund, is "an excellent indicator that this architecture will be a boon to the AI, HPC, and analytics markets."

3 Weeks to 600 Petaflops

This can be scaled up to produce a 600 petaflop supercomputing cluster using 140 DGX A100 systems in just three weeks with the Nvidia's DGX Superpod. The Superpod can be deployed this quickly because it is being offered as pre-configured package, containing nearly 10 miles of optical fiber, and based on Mellanox's InfiniBand interconnect technology.

Nvidia recently closed on its $6.9 billion Mellanox acquisition.

Ampere at the Edge

Nvidia also announced a smaller form factor model for AI workloads at the edge, alongside the heavyweight DGX model.

The EGX A100 is Nvidia's answer to the IoT problem and the need for more compute resources at the edge. Huang explained that because IoT devices and sensors are generating a nearly constant stream of information it is essential that they are located close to the collection point.

"Otherwise the cost of streaming this enormous amount of data is prohibitive," he said. "It's also important to put the data center close by so that you could sense and react to the environment as quickly as possible, where a few milliseconds make a difference."

The EGX A100 combines both an Ampere-based GPU with a Mellanox ConnectX-6 on a single PCIe expansion card.

"By installing the EGX into a standard x86 server you turn it into a hyperconverged, secure, cloud-native, AI powerhouse," Huang said. "It's basically an entire cloud data center in one box."

While the DGX series GPU accelerators are already in full production and shipping to customers, the EGX A100 won't be available until the end of the year.