Earlier this year, SDxCentral explored the market push behind AI inference – the process where a trained machine learning model generates predictions and outputs from new input data.

Dell’Oro Group recently singled out inference requirements for foundational models as helping drive deployments of custom accelerators from hyperscale customers such as Amazon and Google Cloud. This saw global data center server and storage component revenue rise 40% year-over-year in the third quarter of 2025, according to the researchers.

“The dirty secret of AI storage is that most organizations are optimizing for the wrong workload. They're building storage architecture for training, batch-processed, predictable patterns, while inference demands something completely different: continuous operation, thousands of concurrent users, microsecond latencies, and strict economic constraints,” explained Val Bercovici, chief AI officer for storage vendor Weka.

For Weka, inference is where the business value actually gets delivered when it comes to AI.

“Every interaction, every AI agent swarm (think Claude Code, OpenClaw, Manus, etc.), and every reasoning task runs on inference. The economics have shifted: you train once, but you inference millions of times,” said Bercovici. “What's interesting technically is that training and inference are converging through reinforcement learning (RL). DeepSeek's R1 model used roughly 8,000 RL loops, and each loop is fundamentally inference-heavy.”

What may get in the way of these numbers is the compute bottleneck. As AI systems become more memory-intensive for the inference use case, demand for both conventional server dynamic random access memory (DRAM) and specialist high-bandwidth memory (HBM) is dramatically outpacing supply.

According to Weka, storage for model training can't feed graphics processing units (GPUs) fast enough for forward passes. As a result, inference deployments take minutes and hours instead of seconds.

“And when inference workloads exhaust GPU memory, they waste precious cycles redundantly recomputing tokens thousands and millions of times, instead of just once to generate output. Meanwhile, the procurement clock and energy meters are ticking,” Bercovici said.

As such, various AI-native firms have been enjoying a boom in tackling the memory shortage as of late. Take San Francisco-based Tensormesh, which pulled in $20 million in funding from investors including Nvidia, AMD, and the neocloud CoreWeave. The firm specializes in caching-accelerated technology for inference with its inference platform.

Tensormesh peer, FriendliAI, based in Redwood City, California, has also been pulling in the big bucks, having seen funding late last year of a cool $20 million for its own inference solution.

Killer KV

According to both Weka and Vast Data, KV (key-value) caching - which speeds up AI inference by reusing tokens used in a model's previous generations - is compounding the memory deficit as repeated token reuse and intensive agentic workloads inherent in AI inference are rapidly increasing demand for prompt, always-on storage.

“Here's the problem: AI customers aren't buying storage systems to let data sit idle, nor are they using them for backup and archive,” claimed Bercovici. “They're buying storage for continuous AI use cases: training, fine-tuning, and inference, particularly with KV cache running on those systems 24/7. No one shuts off their inference servers at night. These are always-on operations.”

Earlier this year, Vast Data co-founder Jeff Denworth revealed the inference impact on its services when launching a flash reclamation program designed to help its customers overcome the memory shortage.

“Our minimum recommendation for a customer is they need roughly about a terabyte and a half per GPU that they deploy for everything from project storage to database storage, to checkpoint data and data loaders. If we add the KV caching consideration to that, that becomes a 12-times amplification on the amount of network-attached infrastructure that you need.”

Vast espouses a disaggregated approach to the problem where installed flash is converted into a “unified, globally accessible pool,” applying continuous, global reduction across the namespace, helping to eliminate redundant patterns where they occur. Specifically, NVMe drives are reclaimed from legacy arrays, data lakes, and existing servers.

Vast rival Weka, meanwhile, pushes a software-defined architecture for eliminating storage procurement dependency. When customers provision GPU servers to meet their computational requirements, the firm's storage system is deployed on those systems to utilize local NVMe devices for GPU-grade performance.

On the open source side, vendors like Red Hat are touting virtual LLMs (vLLMs), an open source inference platform used within Red Hat products, and which is compatible with hyperscaler solutions.

Red Hat senior principal technologist for AI, Robbie Jerrom, recently highlighted the LLM-D project. Created by Red Hat with Google, Nvidia, IBM Research, and CoreWeave, the venture works in conjunction with vLLM to reduce the cost of inference and improve inference performance via KV cache.

“If you start doing that over time with lots and lots of user responses, you can get cache hits of up to 80-88%, and that drives down cost, but more importantly, it increases the performance, and we can share that cache across multiple models," Jerrom said.

While different, these solutions all make use of existing resources.

“You can't just buy your way out of inefficiency anymore,” explained Bercovici. “Procurement timelines stretch for more than six months, and most GPU clusters still run at 30-50% utilization, which means organizations are sitting on massive underutilized capital.

“Organizations that thrive aren't waiting for supply chains to recover, they're turning what they already have into what they can immediately monetize.”

Bercovici explained that inference relies both on GPU architecture and the quality of the AI model itself.

“But here's what matters in production: model quality determines what you can deliver, whether it’s voice agents, research capabilities, or coding assistants. GPU architecture determines whether you can actually deliver it at scale,” he said.

“You can have the world's best model, but when you scale to thousands of concurrent users, you hit memory and bandwidth limits long before accuracy issues. That's the bottleneck we're seeing: GPUs wasting cycles on token recomputation because KV cache can't stay hot and available.”

The Nvidia factor

That bottleneck may also hit a certain green-colored hurdle, according to Vast’s Denworth, who said Nvidia's recently released approach to key-value caching in the form of Inference Context Memory Storage (ICMS) platform via the chip giant’s BlueField-4 DPUs (data processing units), which he said would exacerbate problems further.

Specifically, the Nvidia stack sees BlueField DPUs as the storage engine handling context movements close to the GPU, while Spectrum-X Ethernet networking keeps access predictable under load. Software orchestrator Nvidia Dynamo, meanwhile, coordinates inference behavior, and ICMS defines context as system memory.

“The alarm bell gets rung when you think about the amount of capacity that they're allocating per GPU. So what is being determined based upon notional context length is you have essentially about 16 terabytes per Vera Rubin GPU, in addition to all the other storage that you already have supporting your AI workloads, both storage and database.”

According to forecasts from Nvidia and analysts cited by Denworth, this brings about a need for 100 exabytes of additional capacity in the market - extra capacity which “doesn't necessarily exist” according to Vast.

Vast’s own operating system was integrated with BlueField-4 DPUs through collaboration between the two firms. In a nutshell, Vast stores, shares, and reuses the context generated by ICMS efficiently across compute pods.

“We basically have gotten to the point now where we can run all of our server code in the actual GPU host in a Bluefield DPU,” claimed Denworth. “This allows us to basically establish a very, very clean line of communication just between GPU memory down to SSDs (solid-state drives), with a single operation that's zero copy all along the data path.”

Vast claims this helps integrate file servers, object storage servers, and database servers into DPUs that already exist in a client’s host, freeing up space for optional data services features.

Independent of Vast and other Nvidia partners, Weka also worked on ICMS, which its AI chief said formalizes treating inference context and KV cache as shared, memory-class infrastructure rather than node-local or transient storage.

Weka’s software-defined architecture is claimed to eschew the need for dedicated storage infrastructure alongside additional hardware which introduces latency and extend deployment timeframes. This shared context system has parallels to ICMS, according to Bercovici.

“Nvidia is continuing to define the future of shared context with ICMS,” he said. “[We help] customers adopt shared context whether they’re ready today or operating within existing architectures.”

Bercovici believes that as agentic AI becomes the norm, driving models to require persistent context across multi-turn interactions, the profitability of enterprise AI hinges on the quality of one’s storage architecture.

“It has to deliver predictable, cost-effective performance at the pace and scale that production workloads demand. Get this wrong, and you've built an incredibly expensive GPU cluster that can't serve customers profitably.

“The question isn't whether you can train a model, it's whether you can serve it economically at scale. That's where most storage architecture breaks down,” he said, before adding: “In the inference era, memory becomes the limiting factor, not compute. Organizations that optimize for inference specifically for context persistence and low-latency memory access will win in 2026.”

This article first appeared in the SDxCentral Memory & Storage Supplement.

To read for free, simply register.