Google researchers have warned that large language model (LLM) inference is hitting a wall amid fundamental problems with memory and networking problems, not compute.

In a paper authored by engineering duo Xiaoyu Ma and David Patterson, the pair contend that while annual sales of inference hardware are projected to soar by up to six-times over the next half decade, the sheer cost of serving state-of-the-art models due to intense high bandwidth memory (HBM) and end-to-end latency requirements may just price out some businesses.

With companies like OpenAI reportedly losing $5 billion annually on inference costs, the research suggests that continuing to optimize for compute performance while ignoring memory and networking constraints is economically unsustainable.

“The current AI hardware philosophy – full-reticle die with high FLOPS (floating-point operations per second), many HBM stacks, and bandwidth optimized interconnect – is a mismatch to LLM decode inference,” the pair wrote.

Memory hits a wall

The core issue the Google engineers highlighted was in how LLMs actually generate text. Unlike training, which processes data in parallel, the “decode” phase of inference is “inherently sequential,” meaning it produces one token at a time in what's called an autoregressive process. This, the pair argue, makes inference fundamentally memory-bound rather than compute-bound.

Further exacerbating issues of memory is the ongoing and ever-worsening memory shortage, which has seen prices skyrocket and producers of HBM like SK Hynix warning of supply chain issues and elongated lead times stretching out until 2028.

But the problem of memory dates back further than late 2025, with the Google researchers noting that from 2023 to 2025, HBM costs increased by 35% for both capacity and bandwidth, even as standard double data rate (DDR) memory costs dropped by roughly half. They argue that manufacturing difficulties stemming from stacking more dies and increasing dynamic random-access memory (DRAM) density are driving HBM prices up just as demand explodes.

The alternative in DRAM has hit a wall of its own, with a fourfold capacity increase that historically took three to six years will now require over a decade, the researchers warned.

They also took issue with static random-access memory (SRAM)-only solutions, like full reticle chips filled with SRAM developed by the likes of Cerebras and Nvidia’s new partner Groq. “While plausible when the companies were founded a decade ago, LLMs soon overwhelmed on-chip SRAM capacity. Both had to later retrofit external DRAM,” the pair wrote.

“Memory bandwidth improves more slowly than compute FLOPS,” the paper reads. “For example, Nvidia GPU 64-bit FLOPS rose by [80-times] from 2012 to 2022, but bandwidth grew only [17-times]. This gap will continue expanding.”

To overcome the memory wall, the two Google engineers propose the use of high bandwidth flash (HBF).

Conceived by SanDisk in February 2025, HBF integrates NAND flash with HBM packaging. The result is a technology that offers up to 16-times the capacity of HBM and delivers comparable bandwidth, all while maintaining a similar price point. That could essentially allow cluster operators to attach terabytes of memory to GPUs. While the researchers note that HBF stacks multiple layers of NAND dies.

“HBF delivers [10-times] memory capacity per node to reduce system size to save power, [total cost of ownership], [carbon dioxide equvalent], and network overhead,” the pair wrote. “The weaknesses of alternatives are bandwidth for DDR5 [double data rate 5], capacity for HBM, and write limits and high read latency for HBF. Another HBF benefit is sustainable capacity scaling; flash capacity continues to double every three years while DRAM growth decelerates.”

Another memory-focused alternative the pair proposed to help boost future AI inference was processing-near-memory (PNM), also known as near-memory computing (NMC), where processing logic is integrated directly to DRAM to overcome CPU-memory bandwidth bottlenecks.

3D stacking techniques, like through-silicon vias (TSVs), see logic layers stacked with memory to boost bandwidth and lower latency. The Google researchers contend that more research should be done into processing-near-memory, as the concept can significantly reduce data movement, latency, and energy consumption for memory-intensive workloads.

The pair noted that despite weaknesses in bandwidth and power when compared to processing-in-memory (PIM), PNM is better suited for LLM inference, writing: “PIM requires software to shard memory structures of LLMs into many small pieces that rarely interact to fit into 32-64MB memory banks; shards in PNM can be [1,000-times] larger, making it much easier to partition LLMs with a low communication overhead.

Latency: The network bottleneck

Beyond memory, Google’s Ma and Patterson argue that the networking assumptions underpinning modern AI infrastructure are fundamentally at odds with the demands of inference workloads, causing misalignments extending beyond mere memory considerations.

For training workloads, traditional data center interconnects (DCI) prioritized bandwidth over latency, with sizable clusters and training supercomputers that would move massive datasets in large batches, while inference runs historically utilized just one chip. But for LLM inference today, the equation is now completely reversed.

“Because of big weights, LLM inference now needs a multichip system, with software sharding that implies frequent communication,” the researchers wrote.

Because the decode phase operates with small batch sizes and generates output token-by-token, they contend that inference workloads produce frequent, small messages where every millisecond of network latency directly impacts user experience – a concept exacerbated by the mixture of expert (MoE) and long sequence models, like DeepSeek V3.2, which increase network hop counts stemming from sizable memory requirements to further compound latency issues.

Add to that long input sequences, retrieval-augmented generation (RAG) database lookups, and especially reasoning models, which generate lengthy "thought" sequences before producing visible output, all amplify latency sensitivity.

As a result: “latency trumps bandwidth for frequent, small messages in a big network.”

To help with latency as well as throughput, the researchers contend the memory solutions outlined above offer indirect networking benefits, such as higher memory bandwidth helping to reduce latency of every decode iteration, while greater memory capacity per accelerator chip shrinks system size, cutting communication overhead.

But Ma and Patterson argue the industry needs to go further by fundamentally rethinking the network latency-bandwidth tradeoff that has dominated data center design.

Among their suggestions for further research were high-connectivity topologies that require fewer hops between nodes such as tree, where devices are connected hierarchically; dragonfly, where the network is divided into groups of high-radix routers, each of which acts essentially as a large, virtual router; and high-dimensional tori, where nodes are connected in a grid that loops back on itself in multiple direction, like an interconnected donut.

While these topologies may sacrifice some bandwidth compared to current designs, the researchers argue that they deliver the low latency that inference workloads desperately need.

Another area of focus that could help provide the latency needed for LLM inference highlighted was processing-in-network (PIN), sometimes referred to as in-network processing (INP).

PIN sees LLM-specific operations offloaded directly into network switches. Commercial implementations already exist, for example, Nvidia's NVLink and InfiniBand switches support in-switch reduction through scalable hierarchical aggregation and reduction protocol (SHARP). And similar capabilities are only just emerging in AI-optimized Ethernet switches.

Finally, and perhaps most intriguingly, the researchers propose codesigning reliability and interconnect to improve both.

They argue that local standby spares can reduce the latency penalty when jobs must migrate after node failures. More provocatively, then, was the suggestion that if perfect communication isn't strictly necessary for LLM inference, systems could reduce latency by using cached or approximated data when message timeouts expire, rather than waiting indefinitely for straggler messages.

“While many researchers explore compute for data centers, we recommend instead improving memory and network along four directions: HBF, PNM, 3D stacking, and low latency interconnect,” the researchers wrote. “Moreover, novel performance/cost metrics that focus on datacenter capacity, system power, and carbon footprint offer new opportunities versus conventional measures.

“Constrained versions of HBF, PNM, PIM, and 3D stacking also might work well for mobile device LLMs. Such advances would unlock collaborative work towards important and urgent innovations that the world needs for delivering affordable AI inference.”