When you think about Kubernetes, clusters of CPU and memory resources all scaling to meet the demands of container workloads probably springs to mind. But where does GPU acceleration fit in this picture?

It’s a question Nvidia has been asking ever since the inception of Kubernetes in 2014. “We've been working with Kubernetes for quite a long time,” said Chris Lamb, VP of GPU computing software platforms at Nvidia, in an interview with SDxCentral.

He explained that Nvidia’s work in this arena has been somewhat drowned out by webscale applications which have been and remain the primary use case for Kubernetes. However, Lamb argues there is a huge potential for GPU-accelerated Kubernetes clusters in artificial intelligence (AI) workloads, an arena where Nvidia has long dominated.

“We actually use Kubernetes internally for our large-scale AI training,” he said. “We’re basically using Kubernetes for high-performance batch scheduling on something that’s basically like a supercomputer with multiple rails.”

Initially, “we started out by working on how you were even going to deal with GPU-accelerated compute nodes in a Kubernetes cluster,” he said.

However, as Kubernetes has evolved, Lamb said Nvidia has shifted much of its focus on furthering the development of SIG Node, which controls the interaction between the container pods running on a node and the host resources that comprise it. According to Lamb, SIG Node is what ensures that a containerized workload running in a Kubernetes cluster can address GPU-accelerated nodes effectively.

Kubernetes isn’t, however, a silver bullet, Lamb said. “Kubernetes provides good API's for resource discovery and management, but it is not the whole solution when it comes to making those resources easy to use.”

Because of this, Nvidia developed Triton, an open source inference serving platform that enables users to deploy AI training models on any GPU or CPU-based interface. By running Triton in a Kubernetes environment, Lamb claims it's possible to completely abstract the hardware from the software. “It’ll work; it’ll auto scale,” he said.

In this case, Kubernetes is acting as the substrate on which Triton runs, he explained. Triton handles the abstraction of the hardware within the node, while Kubernetes orchestrates the cluster, enabling it to scale out more effectively.

Looking beyond software, Nvidia has taken steps to tailor its hardware for use in virtualized environments as well. Though, this isn't specific to Kubernetes. With the launch of the company’s Ampere-based A100 enterprise GPUs and the DGX A100 server, the company introduced multiple instance GPU (MIG).

MIG enables a single A100 GPU to be segmented into seven smaller GPUs, kind of like how a CPU can be segmented into its individual cores. According to Lamb, this enables users to automatically scale their applications using a container runtime like Kubernetes with much greater granularity.

Prior to MIG, each node in a GPU-accelerated Kubernetes cluster would require its own dedicated GPU. With MIG, a single Nvidia A100, of which there are eight in a DGX A100, can now support up to seven smaller nodes, allowing the application and resources to scale more linearly.

Looking to the future, Lamb expects GPUs will begin to the move into the mainstream of Kubernetes, especially as “AI serving becomes a GPU-accelerated workload, which is just at the inflection point of taking off.”

“As things expand, I think most people are going to be able to just think about GPU accelerated as a fast button or an efficient button and not have to think about GPU development or programming,” he added.