Feeding generative artificial intelligence (genAI) models data is a big priority for companies. GenAI is poised to power new applications and dramatically transform the enterprise, but its success hinges on data engineering.
That’s because genAI models are hungry beasts – hungry for data.
For many enterprises, their future competitive edge will be largely based on how well they can wrangle their existing in-house data assets to support genAI and other machine learning (ML) models that may reside on hyperscalers’ clouds.
For AI, the data must be high quality. It must be accurate, relevant, timely, correctly transformed, and available in sufficient volume. This puts pressure on data scientists and data engineers who must quickly learn new computing paradigms.
Many data engineers have been tracking this work as genAI data engineering builds on previous breakthroughs in natural language processing (NLP) and recommendation engines.
Big data days: past innovations paved the way for genAI The course was rocky, but familiar big data innovations like NoSQL, Spark, and Hadoop opened greater amounts of unstructured data for enterprise analytics. That can prove helpful to genAI models, which improve as they are trained and fine-tuned on diverse data sets.
Data warehouses, data lakes, and data lake houses went beyond traditional relational data processing, spawning a new generation of open-source tooling. It could be said that if you have a problem, the Apache Software Foundation has an app for that.
Big data advances include data catalogs that categorize information and add metadata descriptions, as well as workflow orchestration for data pipeline building. Recently, vector databases arose to handle unique storage and retrieval needs of the large language models (LLMs) central to so much genAI work.
But, for frontline data engineers, this means mastering something on the order of a space-age tool box, which is an ongoing process. Moving to a new computing paradigm means learning while working with ML models that can surprise with their results, change day-to-day, and which usually run on GPUs that exhibit styles of behavior – and incur processing costs – new to many.
In genAI today, the data and the model are dancing the tango. Finding success takes both.
AI poses new data engineering challenges Billy Newport, a data engineering veteran with experience at Goldman Sachs and IBM, and who now oversees the DataSurface open-source project on GitHub, observes that while core data engineering principles remain constant, building AI pipelines introduces new challenges.
In genAI, a crucial step often involves transforming corporate data into a format the machine can cope with. That’s a familiar step in all data integration. But here the new twist is vector embeddings, as Newport outlined to SDxCentral.
Fundamentally, genAI works with numbers. Text, images, and other forms of corporate data are best converted to numerical vector embeddings to bridge this gap.
That’s a driver behind new interest in vector databases and vector data enhancements for relational and many other kinds of databases that have grown up in recent years. Now, vector data has crept into the workflows overseen by data engineers.
Newport said a challenge lies in converting things like credit card identifiers – transactional data that is “meaningless to AI” – into something usable.
“Moving data is the same,” Newport said. “Whether it's AI or reporting or something else, you're still moving data. The only difference is that at the end of the pipeline, you might be turning it into an embedding for a vector database, or you might be putting it into a special type of database for financial reporting or whatever. The type of database at the end of the pipeline depends on the use case.”
“There is no such thing as one database that can do everything,” Newport continued. “That's not the way the world works.”
While the requirement to embrace ever newer data types has become familiar, the genAI LLM has its unfamiliar gotchas. The learning experience there is ongoing, as different new development frameworks are regularly updated.
Data engineers preparing data for genAI find there are unique ways in which data is fed to an LLM. And understanding these can reduce cost, which is still one of the major parameters applications are measured.
In the AI-oriented data pipeline, Newport said, ingested data is broken down into context windows. These are essentially smaller chunks of information represented as tokens.
“The size of these context windows can vary depending on the development framework used,” Newport said. “And this directly impacts processing costs.”
It’s well documented that these costs can be high, even if the end user uses a “standard LLM” that is hosted by a major cloud provider.
GenAI data engineering moon shots Guillermo Sanchez Dionis, principal consultant at Xebia Data, contrasted today’s genAI efforts with early ML forays into sentiment analysis. GenAI models are much more complex, requiring significant computing power and engineering knowledge, Sanchez Dionis told SDxCentral.
For most shops, building their own model is like a “moon shot” due to the difficulty involved.
“What is being done most these days is, for example, a bank may have audit rules for a specific ranking and they have a book with 500 or 600 or more rules,” Sanchez Dionis said, noting, “that’s a lot of text.”
Teams index that content in a vector search-style database and then use that to feed an existing model like OpenAI’s ChatGPT or Google’s Gemini with context.
“You use this mixed approach where you provide context data to the model,” Sanchez Dionis said.
Lack of quality data used for this effort can be one of the bigger obstacles to success, which is especially true for unstructured data where there may be no clear definition of quality.
Even semi-structured data, like JSON, can be problematic, Sanchez Dionis said, because the underlying data structure can change over time. And unstructured data quality tends to be dependent on the specific use case.
A takeaway per Sanchez Dionis is that companies should continually work to define clear data quality rules for semi-structured and unstructured data used with genAI models.
Care and feeding of GPUs The hallmark traits of big data – variety, velocity, and volume – all become more acute issues in production with genAI, according to Paco Nathan, managing partner at consultancy Derwen, which is focused on open source integration for AI production.
“The risks associated with incorrect data going into AI apps, either during training, fine-tuning, retrieval-augmented generation (RAG), or various forms of injection attacks at inference time are superheated,” Nathan told SDxCentral. “Basically, the stakes for data engineering are at an all-time high.”
But the benefits of better data engineering are real. Nathan said he has seen teams with frugal means achieve results superior to multi-million-dollar efforts by carefully producing supplemental synthetic data sets that efficiently feed to the AI model.
Importantly, risks emerge as the new GPU hardware paradigm rises. The domination of GPUs in LLMs today can create run-time problems that different engineering groups must sort out, Nathan said. And data engineers may need to develop a feeling for these hardware issues. Data engineers should have some understanding of how genAI models work on GPUs, including the techniques for optimizing GPU performance during training.
“The problem in the way that deep learning training usually goes is that GPUs fail silently,” Nathan said. That may be due to a fault in a summation process where the threads of multiple GPUs do not successfully aggregate work. Nathan said coming advances in GPU connectivity are intended to address this apparent weak point in current systems.
At some point, it is necessary to aggregate underlying neural networks’ results, and, as Nathan puts it, “suddenly it becomes a network engineering problem.”
Basically, debugging errors in deep learning can be difficult. Failures might not show up until later stages, making it hard to pinpoint the cause. Finger pointing – never unknown in software development – may ensue, with some fingers pointed at data engineers. And that can become intense as the cost of genAI computational resources is very high.
“What people are realizing now is like, ‘oh no, this cluster that I'm babysitting, it costs more per day than my annual salary,’” Nathan said. “If I make a mistake, my boss is going to say, ‘Let's get somebody else.’”
Data engineers will have to adapt quickly It is a brave new world for data engineers on the front line of the genAI transformation. Fundamental rules apply, but there are shifts underway that will change some aspects of the discipline.
“One thing that data engineering teams typically need to learn to be effective in this era of machine learning in production is that the large models have a different lifecycle and different risks,” Nathan said.
What might have been standard procedure and organization circa 2020, probably won’t work well today, he continued, urging data engineers “to be curious, learn what’s needed, and especially what’s evolving, then adapt.”
“Data engineering needs to mature,” Newport said. “It’s going to split into a core platform engineering role and also find individual engineers working on different use cases that build on the platform.”
Such thinking lies behind Newport's work on the DataSurface project. One objective: to eliminate engineers’ traditional rote work “building bespoke pipelines for each individual thing they have to do.”
Remember, Newport advises, “what you don't want is everybody reinventing the wheel.”
The role of the data engineer will continue to grow in importance, according to Sanchez Dionis. Even as new challenges arise, he cites basic rules that still obtain results. For genAI applications, data engineers must:
- Perform unit tests on the data transformation logic.
- Implement continuous integration to ensure that changes are non-breaking.
- Make sure you handle data lineage by choosing a tool that handles dependencies between data assets.
Look for the data engineer role to grow in the years to come, even as data platform engineers improve the means of automating pipeline deployment. They must work in informed collaboration with a ML engineer: a person most intimately concerned with everything related to genAI models, as well as new types of AI models as they come along.
Comments