High-performance computing (HPC) hinges on its ability to analyze and categorize large amounts of unstructured data. As the data explosion continues, enterprises are on the hunt for cost-effective storage that can scale with their HPC needs, and according to Cloudian CTO Gary Ogasawara, object storage is the golden goose.

Historically, parallel file systems have been a core technology to accelerate high-performance computing. However, legacy storage performance falls short on delivering insights from emerging workloads such as artificial intelligence (AI), deep learning, and HPC that typically have very low latency and high I/O throughput requirements.

At the center of this challenge, Ogasawara said, is consolidating data, enabling categorization, searching data, and then managing access, which is easier said than done.

These new workloads are driving new architectures and businesses can’t expect to compete in this new data-driven era using old tools developed for an earlier time. Furthermore, the shift to more remote work has exacerbated the need for better visibility and management.

Whether migrating traditional applications, modernizing existing applications, or scaling to meet the performance demands of big compute workloads, object storage shines, according to Ogasawara.

It’s hard to manage data that you can’t see. 

HPC storage infrastructures use traditional data management protocols such as network file system (NFS)  and server messaging protocol (SMB). The primary issue here is that many existing devices and applications don’t support data access over HTTP, specifically via Amazon S3 Application Programming Interface, better known as the S3 API.

The most common “language” of object storage is the S3 API, which is revolutionizing how applications can control data.

Object storage employs the S3 API, so it speaks the language of the cloud. It also incorporates data management features that simplify data placement, cloud and on-prem storage become two parts of a single global namespace, which is revolutionizing how applications can control data Ogasawara said. 

Some vendors are developing interfaces that can store data via traditional protocols and then access that data via S3, but this translation is too slow to meet HPC use case needs.

A modern parallel file system manages data as files in a structure of hierarchical directories technologies. It utilizes a distributed metadata model where metadata is spread evenly across all the storage nodes and each node contains equal amounts of data and metadata.

Object storage, by contrast, manages data as objects, as opposed to files or blocks, and is typically used in the cloud. It’s a hierarchy-free method, meaning it does not use a directory tree like file storage systems. Unlike files, an object can include a large amount of metadata, which enhances search capabilities, Ogasawara said. This makes retrieving data more efficient because there are no bottlenecks created by complex directory systems. 

According to Ogasawara, object storage offers a means to curate and preserve data efficiently for geographically dispersed HPC deployments with a single namespace to support computing systems.

In the face of today’s data deluge, the directory structure is becoming too cumbersome for users to navigate. In this space, object storage could truly thrive. By this account, object storage is ideal for bringing structure to unstructured data such.

Projections from IDC show that 80% or more of global data will be unstructured by 2025. The pressure is on for IT teams looking to store and manage data in a way that can be easily accessed with speed and consistency.

The problem of files is a problem of scale. If more storage is needed, traditionally the answer has been to buy more servers. Not only is the cost of this approach wildly expensive, but the time it would take to scale to the level required is inefficient. Even if organizations had unlimited time and money to add many new servers, a problem remains. Vertical storage architecture contains bottlenecks that slow performance to a near-crawl. 

“It's harder to scale parallel file systems and file systems in general,” Ogasawara explained.  “As the amount of data that HPC applications want to bring to bear increases to petabyte, exabyte level they really are sort of trapped because it's hard to build out a parallel file system to that level of scale. So they're doing a bunch of strange contortions like having to silo data and having multiple different different storage systems.”

Object storage tackles the scalability challenge that hinders traditional infrastructure appliances in HPC workloads. An object storage cluster expands in a linear fashion. Nodes can be added to the cluster as needed without needing to plan capacity increases months in advance or pay for unused.

This is the advantage of object storage’s single namespace, Ogasawara said. “Whatever data the application wants, wherever it is, it just can address that data and access it.”