Just in time for Halloween, Databricks this week took aim at the swamp monsters lurking in data lakes, sacrificing its open-source software project Delta Lake to the open-source overlords, flying under the Linux Foundation’s open governance model flag. 

The announcement was made yesterday at the Spark+ AI Summit in San Francisco and follows a $250 million Series E funding round in February, bringing Databricks’ valuation to $2.75 billion.

The big data firm launched the Delta Lake Project in April as a storage layer to live on top of Spark SQL and Parquet files stored in Databricks File Systems within data lakes to manage large sets of data. 

As the industry shifts towards the adoption of open data lakes with open and extensible access to data without vendor lock-in around proprietary data formats, systems, or APIs, “companies don’t want their strategic data lake data, which holds key operational data for their entire company, to be locked up by vendors, and are increasingly aligning around open solutions,” a Databricks spokesperson told SDxCentral.

"Open source and open-format access also means that the data can be read by different readers, tools, and vendors, providing more flexibility, extensibility, and value for data lakes," the spokesperson added.

Delta Lake has seen significant growth since joining the open-source community earlier this year, with more than 4,000 organizations using it, and over 2 exabytes of data processed in just the last month, according to the Databricks spokesperson.

“We’re excited to foster even more growth by partnering with the Linux Foundation. We are joined by Alibaba, Booz Allen Hamilton, Intel, Starburst in the announcement to develop Delta Lake support not just for Apache Spark, but also Apache Hive, Apache NiFi, and Presto,” the spokesperson added. 

Data Lakes possess an innate capacity for open mindedness, permitting any and all data — structured or unstructured in native formats — an unobstructed entrance into its waters. By housing raw and transformed copies of source-system data from reporting, visualization, analytics and machine learning, it's a great tool for discovering new insights, according to Databricks.

But with free admittance, unwanted visitors are guaranteed entry and that leaves data lakes vulnerable to becoming data swamps.

By offering anyone within the organization access to data, large organizations in particular are susceptible to unwanted swamp monsters like failed writes, schema mismatches, and data inconsistencies as a result of unreliable or inaccurate information. 

Delta Lake was designed to siphon out these unwanted visitors to increase the reliability, quality, and performance of information in data lakes. It does this by managing transactions across batch and streaming data and multiple simultaneous writes that also remove the need to build the complicated data pipelines that are used to move information across different computing systems.

Delta Lake deploys atomicity, consistency, isolation, durability (ACID) transactions, and brings in serializability — the strongest level of isolation — to ensure data in the data lake is usable for downstream data science, machine learning, and business analytics, even in the event of errors and power failures.

Along with ACID transactions, to prevent data lakes from becoming data swamps the platform's key features also include:  

• Scalable metadata handling with Spark's processing power to process petabyte-scale tables with billions of partitions and files.
• Time travel (data versioning) takes snapshots of data and provides the option to revert back to earlier versions for audits, rollbacks or to reproduce experiments. 
• Unified batch, streaming, source, and sink for historic backfill and interactive queries. 
• Schema enforcement that allows for schema specification prior to enforcement. 
• Schema evolution automatic updates for table schema changes. 
• Audit history records details about every change made and creates an audit trail. 
• Updated delete and merge Data Manipulation Language for data lifecycle management in compliance with General Data Protection Regulation (GDPR) and the California Consumer Privacy Act (CCPA) to simplify use cases like change data capture. 
• 100% compatible with Apache Spark API, the commonly used big data processing engine supports existing data pipelines.  

Databricks claims these features are not available from any of the existing Linux Foundation projects.