Enterprise IT teams looking to deploy large language model (LLM) and build artificial intelligence (AI) applications in real-time run into major challenges. AI inferencing is a balancing act between three main factors: throughput performance, latency and cost-effective deployment.

With 97% of overall GPU revenue in 2023, Nvidia is the market’s dominant player. Fortunately, the chip giant’s technology has proven worthy of the hype.

Nvidia’s accelerated computing platform broke performance records on the two new generative AI (genAI) workloads tested in the latest MLPerf Inference benchmarking.

MLPerf measures inference performance for different deep-learning use cases with peer-review and industry-vetting processes. MLPerf Inference v4.0 included a (LLM) benchmark based on the Meta Llama 2 70B LLM and a text-to-image test based on Stable Diffusion XL.

The testing marks the inaugural benchmarking of Nvidia’s H200 Tensor Core GPU, which raised the bar for performance in both of the new test cases. Specifically, H200 is 45% faster than H100 when inferencing a Llama 2 70B model.

With more memory bandwidth than its predecessor, “H200 delivers even more performance versus what we have with H100,” Nvidia director of Accelerated Computing Products Dave Salvator told reporters yesterday.

H200 boasts 141 GB of HBM3e memory with 4.8 TB/s of memory bandwidth. That translates to 1.4-times more GPU memory bandwidth than the H100.

“By having that additional memory, we can not only fit the entire model in a single GPU – which we could do before – but now we have even more room for that KV cache, which along with other optimizations has helped us get this great performance on H200,” he said.

Nvidia H200 also incorporates custom thermal design that increases performance by 14%. “We were able to basically dial up the [thermal design power] TDP a little bit and achieve better frequency residency,” Salvator said.

H200 is sampling now to customers; shipping in the second quarter

Nvidia H200 GPUs are now sampling to customers and will be shipping in the second quarter of 2024. H200 will be available from leading cloud service providers (CSPs) and OEMs like Microsoft Azure, Google Cloud, Amazon Web Services (AWS), Oracle, Hewlett Packard Enterprise, Dell Technologies and Lenovo.

“It's a pretty wide list of folks who are going to be building systems and building instances based on H200,” Salvator said. “It’s a testament to the extensibility of the Hopper architecture that we continue to innovate with.”

Nvidia also submitted its high-performance, open source software library TensorRT-LLM for MLPerf testing and found the software nearly triples LLM inference performance on Nvidia GPUs.

Specifically, H100 Tensor Core GPUs using TensorRT-LLM were 2.9-times faster with the GPT-J LLM compared to previous rounds of submissions. TensorRT-LLM was also key to Nvidia’s high performance on the Llama 2 70B test.

Nvidia credits these successes to key features of the TensorRT-LLM software like inflight sequence batching, paged KV cache, tensor parallelism and quantization.

Inflight sequence batching increases GPU use during AI inferencing; paged KV cache optimizes memory consumption; tensor parallelism lets IT teams split weights across GPUs and nodes to support efficient inferencing at scale; and quantization reduces model size and augments performance.