Cloud data warehouses are an improvement from the legacy on-premises versions, but they’re still just data warehouses, according to Tomer Shiran, co-founder and CEO of data lake engine company Dremio. Shiran says the cloud crusades will escalate this year, particularly in the realm of modern open cloud data lakes, as big data adoption continues to explode. 

The maturation of the technology stack, in addition to more machine learning frameworks entering the mainstream, has both accelerated cloud data lake adoption and sparked an evolution on two fronts: open cloud data lake storage and proprietary cloud data warehouses. “We believe the former will eclipse the latter,” Shiran said.

Data lakes permit any and all data — structured or unstructured in native formats — an unobstructed entrance into their waters. By housing raw and transformed copies of source-system data from reporting, visualization, analytics, and machine learning, they currently sit as an untapped pool to new insights.

Today, more data crosses the internet every second than was stored in the entire internet just 20 years ago, and how that data is collected, accessed, and analyzed can determine whether a business sinks or swims. 

“Savvy enterprises have figured out that cloud data warehouses are just a better implementation of a legacy architecture, and so they’re avoiding the detour and moving directly to a next-generation architecture built around cloud data lakes,” Shiran said in his 2020 predicitons. 

Historically, enterprise analytics infrastructure leaned on proprietary data warehouses deployed as appliances. However, those on-premises services tightly coupled storage and compute inside the appliances and made them hard to scale. 

Mass migration to the cloud brought about a new generation of proprietary data warehouses implemented natively in the cloud. And while they are a step-up from the legacy on-premises versions, according to Shiran, they still represent the same proprietary architectural model where enterprises do not own and do not control that storage.

Open cloud data lake storage, on the other hand, has emerged as the next-generation big data platform, superseding and replacing legacy on-premises Hadoop deployments, Shiran said. 

“Because the storage is in the cloud, it is now easy for enterprises to bring cloud services to that data, without needing to copy or move it,” he added. “As a result, we see the emergence of best-of-breed cloud compute solutions, able to work directly against the data lake storage. This new architecture truly separates compute and storage, and we believe it will emerge as the dominant architecture.”

The use of data lakes has grown alongside the increased use of cloud platforms as storage mediums. Gartner released a report last year that predicts 75% of all databases will be deployed or migrated to the cloud by 2022.

Organizations are developing and deploying new applications in the cloud and moving existing assets at an increasing rate. And this, according to the report, is expected to grow in response to businesses need to extract in-depth insights from growing volumes of data and access data from departmental silos, mainframe, and legacy systems.

This is further underscored by additional market research that projects the global data lake market size will grow from $7.9 billion in 2019 to $20.1 billion by 2024.

“We also believe this begins with systems for data management solutions for analytics (DMSA) use cases — such as data warehousing, data lakes, and other use cases where data is used for analytics, artificial intelligence (AI) and machine learning (ML),” said Donald Feinberg, distinguished research vice president at Gartner, in the report. “Increasingly, operational systems are also moving to the cloud, especially with conversion to the SaaS application model.” 

Dremio, which launched in 2015 by former MapR employees Shiran and Jacques Nadeau, developed an analytics service called Data Lake Engine that aims to deliver fast query speeds and a self-service semantic layer that operates directly against data lake storage. 

This is done by connecting directly to physical data sets such as Amazon Web Services (AWS) S3 and Microsoft ADLS — meaning customers don’t have to send their data to Dremio or have it stored in proprietary data warehouses — and then providing SQL access to any combination of those sources to various data consumers. 

According to Shiran, Dremio enables business analysts and data scientists to explore and analyze any data in a self-service fashion at any time, regardless of location, size, or structure, using their preferred tools such as Tableau, Python, and R. Dremio leverages technologies to deliver live, interactive query performance, including Apache Arrow; a patented data acceleration capability called Data Reflections; Columnar Cloud Cache (C3) and Predictive Pipelining for high performance reads; and highly optimized native push-downs.

As the industry shifts toward the adoption of open data lakes and away from vendor lock-in around proprietary data formats, systems, or APIs, then cloud data lakes combined with open source will “create an opportunity for a new architecture that truly separates storage from compute,” Shiran said. “This enables organizations to own and control their storage, and to bring best-of-breed compute solutions to work with their data as business requirements demand.” 

Open-source tools are key to delivering data lake functionality. And while adoption of cloud data lakes is already rapid, to achieve the best return on investment for big data systems, companies need to marry commercial tools with open-source components to manage complexity to deliver scalability, he added.

Dremio is built on open source technologies including Apache Arrow and Apache Arrow Flight, which the company co-created to provide columnar, in-memory data representation and sharing. 

The company claims Arrow is now the de-facto standard for in-memory analytics, with more than 1 million downloads per month — a number that Shiran predicts will reach 10 million downloads per month in 2020.

Apache Arrow Flight, another co-creation from Dremio that was released last October, extends the performance benefits of Arrow to the Remote Procedure Call (RPC) layer to increase data interoperability. Although the Arrow Flight is relatively new to the market, Shiran predicts it will replace decades-old ODBC/JDBC as the de facto way in which all modern data systems communicate by 2025.

“We believe the addition of transactional tables via the Apache Iceberg project will accelerate it even faster,”  Shiran explained. “Apache Iceberg is an open table format for huge analytic datasets, and it will enable all data warehousing-type workloads to be handled directly by cloud data lakes.”