An open standard for AI inference backed by Google Cloud, IBM, Red Hat, Nvidia and more was given to the Linux Foundation for stewardship in further proof training has been superseded by inference in the AI race.

Announced in time for KubeCon Europe, the Cloud Native Computing Foundation (CNCF) accepted LLM-D, styled as llm-d, as an official Sandbox project. The move was positioned as leading both the evolution of Kubernetes and the broader CNCF landscape into AI infrastructure.

LLM-D was launched in May 2025 as a collaboration between Red Hat, Google Cloud, IBM Research, Nvidia, and neocloud CoreWeave to drive inference through “any model, any accelerator, any cloud.” The project was backed by the likes of AMD, Cisco, Hugging Face, Intel, Lambda, and Mistral AI, as well as gaining academia support from the University of California-Berkeley, and the University of Chicago.

The standard is based on a pre-integrated, Kubernetes-native distributed inference framework and looks to bridge the gap between high-level control planes such as KServe, which serves machine learning (ML) models with standardized protocols for both predictive and generative AI, and low-level inference engines like virtual large language models (vLLMs). Its three tenets include inference-aware traffic management; implementing the Kubernetes Gateway API Inference Extension (GAIE); and utilizing the Endpoint Picker (EPP) for programmable, prefix-cache-aware routing.

Native Kubernetes orchestration is another fundamental, leveraging primitives such as LeaderWorkerSet (LWS) to manage complex multinode replicas and wide expert parallelism, helping turn AI infrastructure into manageable cloud-native microservices.

LLM-D also addresses the the asymmetry between resource and utilization through prefill and decode disaggregation, with prompt processing and token generation phases disaggregated into independently scalable pods.

“By introducing model- and state-aware routing policies that align request placement with specific hardware characteristics, LLM-D maximizes utilization and delivers measurable gains in critical inference metrics like time-to-first-token (TTFT), time-per-output-token (TPOT) [and] token throughput,” CNCF noted of the project.

The foundation also highlighted benefits regarding key-value cache (KV cache), a token cache that registers prior interactions with a model to avoid activating the GPU when such interactions are repeated.

CNCF highlighted LLM-D gains such as in a multitenant software-as-a-service (SaaS) use case, claiming that prefix caching via its inference scheduling in such shared customer maintains near-zero latency and massive throughput when compared to a baseline Kubernetes service.

Future plans for the framework include work with the CNCF AI Conformance program to ensure essential capabilities like disaggregated serving are interoperable across the ecosystem, and more collaboration with open-source AI hub and the PyTorch Foundation.

As covered in the inaugural issue of SDxCentral magazine this week, we are now seeing the breakout of AI inferencing, the process where a trained ML model generates predictions and outputs from new input data. As AI models are now designed to run continuously, models are no longer running separately across training and inference modes.

The likes of Red Hat confirmed its use of LLM-D to reduce the cost of inference and improve inference performance as using KV cache.

“If you start doing that over time with lots and lots of user responses, you can get cache hits of up to 80-88% and that drives down cost, but more importantly it increases the performance and we can share that cache across multiple models," Robbie Jerrom, senior principal technologist for AI at Red Hat, told this title.