IBM worked with Nvidia and Samsung to demonstrate a content-aware storage (CAS) system that can hold a 100-billion-vector database on a single server, work targeted at making retrieval-augmented generation (RAG) easier to deploy directly alongside enterprise data.

RAG deployments rely on separate vector databases, storage systems, and GPU-accelerated ingestion pipelines. As vector counts grow into the hundreds of billions, indexing time, reindexing, and infrastructure cost become major obstacles, with scaling particularly challenging.

IBM Research’s approach pushes document vectorization and related processing into storage itself, reducing the need for separate infrastructure layers around enterprise data. The system understands the meaning or structure of the data it stores rather than treating everything as raw files or blocks.

Under the hood

IBM’s system uses decoupled compute and storage, GPU acceleration for indexing, and a hierarchical index design where vectors are organized in multiple layers so that only a subset of data is searched at query time. Decoupling was achieved using the IBM Storage Scale high-performance Elastic Storage System (ESS), described as an all-flash appliance for AI, high-performance computing (HPC), and massive data workloads.

A total of six Nvidia H200 graphic processing units (GPUs) were used to accelerate rebuilding of individual indexes, reportedly reducing central processing unit (CPU)-based index building from hours to minutes on GPUs.

Samsung supplied 48 high-capacity enterprise NVMe SSDs as built on its latest-generation TLC V-NAND flash memory. This is a triple-level cell stack of Samsung’s proprietary 3D non-volatile (NAND) flash architecture, with each V-NAND drive offering up to 30.72 terabits (TBs) of storage, up to 12,000 MB/s sequential read speed, and up to 6,800 MB/s sequential write speed.

IBM Research said it scaled a prototype to 100 billion vectors with more than 90% recall precision and under 700 milliseconds mean query latency. In tests using synthetic data, loading and indexing took 13 days total, versus an estimated 120 days on a two-socket Intel CPU. Meanwhile, the total data footprint on storage as made up of vectors and index was 153 tebibytes (TiB).

“The team performed experiments to measure query latency and recall precision, the latter of which was achieved by using brute-force search to extract ground-truth from the massive dataset – which itself took many days to perform. The result: a mean query latency of 694 milliseconds with a recall precision of 90%,” IBM reported.

IBM and Nvidia are working toward indexing more than 100 billion vectors in a day, cutting ingestion time, and reducing search latency further through the use of Nvidia’s CUDA Vector Search (cuVS).

Vendors like Alibaba Cloud have experimented with embedding vectorization into storage, with the Chinese giant last year unveiling an AI-powered feature designed to provide large-scale vector data storage and retrieval. The so-called Vector Bucket offering unifies raw and vector data management in its Object Storage Service, accessible via standard APIs to simplify RAG processes.

Chinese player DeepSeek this year demonstrated more general efforts in offloading or compressing AI memory workloads to relieve GPU pressures with a concept that offloads static knowledge from the primary memory of a large language model (LLM) to host memory (CPU RAM). This is designed to relieve the model’s core computational network by saving billions of parameters.

A more headlining approach is Google's TurboQuant, an advanced compression algorithm designed to significantly reduce key value memory.