Data is every organization’s lifeblood, fueling almost every strategic decision.

And, while most modern enterprises use some type or blending of data warehouses, data lakes and data lakehouses, these have limitations when it comes to observability.

Data and artificial intelligence (AI) platform company Databricks says the next-level iteration of all this is the Data Intelligence Platform (DI) powered by AI. This new type of intelligent system can truly understand an organization’s data, the company says, rather than just aggregating and analyzing it.

Databricks recently announced what it calls an industry-first DI, now in private beta. It is the latest in a series of announcements and innovations — including a new partnership with AI/ML platform company Dataiku — from the 10-year-old company last valued at $43 billion and now floating the idea of an IPO.

“Historically, data platforms have been hard for end-users to access and for data teams to manage and govern,” a Dataiku spokesperson told SDxCentral. “Data intelligence platforms are set to transform this landscape by directly tackling both these challenges — making data much easier to query, manage and govern.”

Data lakehouses to data intelligence

As data continues to accumulate into the exabytes, zettabytes — and some say even beyond measurability — organizations have increasingly adopted and managed data warehouses and data lakes, which respectively aggregate and process and analyze data.

Building on this, Databricks introduced the idea of the data lakehouse five years ago. This merged platform queries all data sources in an organization and governs the workloads — such as business intelligence (BI) or AI tools — that make use of it.

While this concept helped elevate data governance, existing tools still have their challenges, according to Databricks. This includes complexities in data accuracy, management and curation; technical skill barriers (querying data requires skills in SQL, Python and BI); governance and privacy; and the high demands of emerging AI applications.

This is because data platforms do not “fundamentally understand” data and how it is used, Databricks says. But generative AI is taking a lead here; in fact, the company says that “AI will eat all software,” meaning that software will become more intelligent as it ingests and processes more and more data.

Offering ‘deep understanding’ of data

Data Intelligence Platforms build on the lakehouse idea by automatically analyzing data and how it is used (reports, lineage, queries and so on) to add new capabilities.

Users don’t have to have technical skills: With genAI, they can ask about data and receive insights powered by large language models (LLMs) from MosaicML.

Databricks claims that this level-up ability can enable such functionalities as tailored natural language access; automated management and optimization; enhanced governance and privacy; and semantic cataloging and discovery (genAI can understand data models, metrics and KPIs to offer new features or identify discrepancies in data use).

BI tools, while important, only have a glimpse into the majority of workloads, thus limiting their level of semantic understanding. This is why natural language querying capabilities have not yet seen widespread adoption, Databricks says.

“DI platforms offer a deep understanding of data and its use, which will be a foundation for enterprise AI applications that operate on that data,” according to a Databricks spokesperson.

DI tools will be a “cornerstone” for intelligent organizations, enabling them to quickly create quality, next-gen data and AI apps.

“As AI reshapes the software world, we believe that the leaders in every industry will be those who leverage data and AI deeply to power their organizations,” Databricks says. “We believe that data intelligence platforms will greatly simplify the development of enterprise AI applications.”

Data platforms ripe for innovation through AI

This evolution to the Data Intelligence Platform arose out of Databricks’ acquisition of MosaicML for a staggering $1.3 billion that CEO Ali Ghodsi called a “bargain.”

Databricks explained that the company has leveraged that acquisition to generate AI models in a Data Intelligence Engine it calls DatabricksIQ, which fuels all parts of its platform. The new Mosaic AI offers multiple capabilities to directly integrate enterprise data into AI systems, the company explains, and will make it easier for enterprises to create AI applications that understand their data.

A company spokesperson said that the platform is currently in private beta, but they will soon announce its general availability and example use cases.

“We believe that AI will transform all software, and data platforms are one of the areas most ripe to innovation through AI,” Databricks told SDxCentral.

Databricks-Dataiku partnership

Generative AI (genAI)  and large language models  (LLMs) are arguably the most transformative technologies to come along since the cloud.

Still, because they are so complex and evolving so rapidly, many enterprises are struggling to adopt them efficiently, effectively and safely.

To help accelerate this process, Dataiku added Databricks to its LLM Mesh Partner Program.

The partnership will help simplify adoption on the Dataiku platform and Databricks’ Lakehouse infrastructure, providing direct access to data in Databricks’ Delta Tables, according to the company. Enterprises will also be able to connect to Databricks-hosted LLM models (including MosaicML) to use in Dataiku’s Prompt Studios, visual LLM recipes and Retrieval Augmented Generation (RAG).

LLMs are the best information-extraction service since Google, able to access “tons and tons of information” and rapidly perform nearly every type of summarization or classification, Jed Doughtery, Dataiku’s VP of platform strategy, told SDxCentral.

“Investigatory assistance work becomes quite easy with LLMs,” said Dougherty. “These librarian-esque tasks are areas where LLMs are under-utilized now but will become ubiquitous.”

A unifying mesh layer

Applying LLMs to large-scale business problems is complicated because organizations need an underlying model provider, a storage capability, a user interface layer and a strong level of security, Dougherty said.

The 10-year-old Dataiku serves as a unifying mesh layer that sits on top of Databricks, connecting to the underlying LLMs and providing governance and connectivity, he said.

“Dataiku is a tool on top of really complicated tools,” he said. He emphasized that, “we don’t have our own LLMs, we want you to be able to use the best class ones out there.”

As organizations increasingly make use of multiple LLMs for different use cases and workflows, the LLM mesh provides a central location to keep track of them all. Dougherty pointed out that LLMs need to clean up, join, prepare and organize data, apply it in batches and automate workflows.

Because “you still have all the same problems you’ve always had with data,” he said. “The existing processes of dealing with data are not going to go away.”

Think beyond the chatbot — far beyond

While new enterprise use cases for LLMs and genAI arise — and are reported on — every day, many companies continue to struggle to understand that the technology is not just chatbots, Dougherty noted.

While that’s a good start, organizations also need to be thinking about LLMs as they can be applied to any existing natural language processing (NLP) workflow. Models can be applied en masse to large datasets, which “really speeds up immediate applicability and capabilities,” he said.

“It’s amazing that you can apply LLMs to super classic machine learning (ML) problems and get amazing results,” said Dougherty.

This includes classifications, “old-school regression,” deep learning and image recognition. As LLMs get better at multimodal workflows, more advanced capabilities will include such tasks and image and video processing, he pointed out.

“Every C-level across the entire world wants to see a chatbot for their organization right now — that’s the minimum, that’s the base,” said Dougherty. “They’re now beginning to realize that every single non-structured piece of data can be applied to LLMs to extract information.”