The real challenge of scaling AI agents isn’t inference, it’s memory. As fleets scale from hundreds to tens-of-thousands, it’s the cost of retrieving, storing, and maintaining human-like knowledge that is becoming the next architectural and economic hurdle.

The AI cost crisis, however, doesn't lie with the models themselves, it lies with the data. Teams spend a lot of time comparing token prices, testing smaller models and tuning prompts. Those changes can cut the model bill, but the underlying systems may still be doing a large amount of expensive work.

Data fragmentation

Most large organizations will have a relational database for transactions, a document store, a graph database, a search engine, a vector store, and several extract, transform, and load (ETL) jobs keeping them in step. Because each database solves a specific problem, data arrives sequentially, type-by-type. The trouble starts when agents have to use several data sources to accurately answer one question.

For example, a customer support agent may need to span multiple systems just to gather the relevant data for a simple query. The agent then compares those results and decides whether it has enough context. If it doesn’t, another round of retrieval starts. All of this creates latency and consumes compute before the prompt reaches the model. 

Overpaying for uncertainty

Published research on retrieval and long-context use, together with prompt-engineering guidance from OpenAI and Anthropic, show token usage spirals because teams often compensate for uncertain retrieval by sending more context than the model needs. Instead of sending the single most relevant 400-token knowledge-base article, a system might send the top-10 search results, three versions of the same policy from different systems, surrounding just-in-case sections, and a lot of unnecessary conversation history. The model then has to sort out what is relevant, which quickly turns a few hundred tokens into several thousand.

The operational bill is broader still. Each database has its own deployment model and associated procedures and skills. The ETL between them has to remain accurate as source systems change. It’s common to see architectures where every component made sense when it was introduced, yet the connections between them have become a spaghetti-like mess.

Persistent memory at scale

The third cost driver is more existential. How do we store long-term memory that changes and builds in real-time? This is the next frontier in large-scale agent deployments and requires a total rethink of storage away from simple retrieval towards human-like understanding. Before we get there, however, there's a more practical question: can we reduce costs by simplifying the data infrastructure? This brings us back to database consolidation.

Reduce the number of handoffs

Traditional stacks separate online transaction processing (OLTP), analytics, and search, but agentic AI collapses these boundaries. Agents need durable read-write interactions, real-time triggers, and low-latency retrieval-augmented generation (RAG) across text, vectors, and graph data. Separate stores introduce latency and duplication and risk pushing semantic retrieval and policy closer to operational data.

Database consolidation is an obvious answer, but wholesale replacement of every specialist database isn’t always the right path. Specialist databases exist for good reasons, and migrations carry cost and risk. But, AI agents struggle when several databases hold different representations of the same information, or when agents are forced to pull partial answers from each one and reconcile them in code.

This kind of consolidation is already being seen in production environments. Where companies have combined several databases – Verizon, Tencent, and Samsung Ads are good examples – they see average response times and training costs slashed, and operational costs significantly reduced. Fewer systems mean fewer copies to synchronize, fewer result sets to combine in the application layer, and less infrastructure to operate.

Agent memory changes the write pattern

As agents work, they record tool calls, plans, permissions, intermediate results, and changes in state. That record is the provenance they need to operate accurately, and it has to stay updated and retrievable in real time.

In agent fleets of thousands, this exposes significant limits in storage architectures, which are usually built for simple retrieval around a single write leader. Read replicas can add retrieval capacity, but writes still pass through one coordinating node. Adding more machines does not necessarily increase write throughput if they all queue behind the same leader.

In experimental multiwriter designs, many nodes take part in processing writes. Researchers at the University of Washington's Systems Lab were among the first to show this is both viable and cheaper. Their transactional application protocol for inconsistent replication (TAPIR) commits transactions in a single round-trip without central coordination. And in their tests TAPIR's key-value store recorded around 50% lower latency and more than three-times the throughput of conventional transaction protocols. 

Storage engines using TAPIR remove the single-write-leader bottleneck for large agent fleets. Engineers can currently scale nodes to 15, each accepting writes at the same time, running on top of multipetabyte scale object storage. 

Everything everywhere all at once

There’s also the thorny question of provenance. If a million agents need access to the same memories, and each agent creates and modifies memory over time, we’re looking at devoring petabytes of storage. Recent context may be queried constantly, while months of old conversations and decisions are rarely touched. Keeping all of it on high-performance storage is prohibitively expensive.

Historical state introduces a major trade-off. Store every version of a record in full and capacity requirements rise quickly. Store only the differences and reconstructing an old state may require replaying hundreds or thousands of changes. 

Periodic snapshots, combined with smaller deltas between them, can balance storage cost against retrieval time. Separating compute from durable storage helps too. Active data remains close to the compute layer, while older state sits in lower-cost object storage until it is needed. Compute can then scale with current demand rather than with the total volume of accumulated memory.

Account for every step

Model pricing is easy to compare because it’s charged in units. By contrast, the cost of an agent task is spread across systems required to execute it, from database queries and network calls to data processing, memory and service availability.

As a starting point, teams should trace one representative workflow from the user request to the final response and account for every step. That exercise often reveals repeated retrieval, overlapping indexes, stale copies of the same data, and context that is rebuilt for every request.

Token optimization still belongs in that review. Once organizations measure what happens around the model, they often find that a large share of the cost comes from moving, reconciling, and repeatedly reconstructing data. In many enterprise deployments, that is where the next meaningful savings are waiting.