AWS is introducing a preview feature that enhances Amazon Simple Storage Service (Amazon S3) by enabling automatic metadata generation when objects are added or modified. This feature aims to address the challenges customers face in managing large-scale S3 buckets, which can contain billions or trillions of objects.

Customers will benefit from capturing detailed metadata without needing to build complex systems. The metadata will be stored in managed Apache Iceberg tables, allowing users to utilize Iceberg-compatible tools such as Amazon Athena, Amazon Redshift, Amazon QuickSight, and Apache Spark to query and retrieve data efficiently.

The new metadata schema includes over 20 elements such as bucket name, object key, and encryption status, among others. Users can input additional application-specific data, which can be linked to the existing metadata during queries.

How it works Users can start capturing metadata by designating a specific S3 bucket for storage. Upon object creation, update, or deletion, related metadata will be recorded in the designated table almost immediately. The tracking includes a timestamp and a sequence number for each event.

Enabling and Querying Metadata To set up the feature, users can create a metadata table bucket via the AWS Management Console or using command line instructions with AWS CLI. Queries can then be executed against the metadata table to retrieve historical records or specific updates.

To further facilitate testing, users can utilize tools like Apache Spark for querying the S3 metadata. The Spark application can reference the Amazon S3 Tables Catalog for easy integration.

Management of metadata configurations is also streamlined through the Amazon S3 Console. By accessing the Metadata tab, users can set up and adjust their configurations conveniently.

Available Now This preview feature is available in the US East (Ohio, N. Virginia) and US West (Oregon) AWS Regions. Alongside this, integration with AWS Glue Data Catalog for enhanced querying and visualization capabilities is also in preview.

Pricing will be based on the number of updates made and include storage costs for the metadata table. Detailed pricing information is available on the S3 Pricing page.

This new functionality promises to empower customers in managing their S3 data more effectively, allowing for insightful data analysis and applications in various fields.