Natural language processors like ChatGPT have become a key tool in the mission to automate data analytics. But these processors are still in their early days, and at this stage should be treated much like “interns” that are not yet ready to take the place of high-level analysts, said Alon Goren, CEO and founder of AnswerRocket, an artificial intelligence-powered data analytics provider.

Traditional data analysis has been akin to searching for information on Google, Goren told SDxCentral. Google is “very efficient and fast” at retrieving answers from websites where the information already exists – it's just a matter of finding it.

Similarly, in the case of data queries, often the answer is sitting somewhere in a spreadsheet and it’s up to analysts to turn data into business intelligence. The challenge is that the average enterprise has hundreds of data dashboards or spreadsheets they need to search, Goren said.

This makes it more difficult and time-consuming to answer questions that are more open-ended and have to do with more than one data point, rather an analysis of a series of numbers. “Most of the complication is about being able to translate data into something that feels like business-related information,” he added.

The typical analytical flow requires analysts to run a variety of queries, look at a variety of data sources and then come back with some kind of summary, which usually prompts a follow-up conversation.

“Some of the analysis could take weeks, so by that point you may lose momentum on the issue,” Goren said.

This is why analytics providers like AnswerRocket are using natural language models like ChatGPT to automate that process. “We need ChatGPT to act as an analyst,” Goren explained.

Natural language models for narrative data analytics

Natural language models like ChatGPT are ideal for use cases where it's not enough to retrieve crude data, but instead requires the type of assessment and presentation an analyst would provide.

Newer language models don’t just transform information into charts or bullet points – which until now were considered state-of-the-art capabilities, Goren pointed out – but rather introduce the concept of “writing a narrative” for users.

Models like ChatGPT use underlying data analytics infrastructure to write “human-caliber narratives, granting businesses the ability to make a decision during a meeting versus saying 'let's follow up. We need a second meeting and third meeting,'” Goren said.

And it’s not only a narrative, but what takes the shape of a two-sided, anthropomorphic conversation.

“In the historical natural language processing world, there were limited tools that you could use to be effective in terms of understanding the user question,” Goren said. Before, even a simple typo could impair a language processor's ability to respond.

“Large language models [LLMs] have changed that as they don't care what language you're speaking. You can misspell things, as long as I can generally get the gist of what you're asking,” he added.

ChatGPT concerns

Although they are “growing intelligence pretty fast,” Goren warned that users should be aware that these natural language processors are capable of hallucinating information. ChatGPT is a writer of fiction and nonfiction mixed together. It can choose either of those roles depending on what it thinks is going to give users the best “human readable answer.”

Processors like ChatGPT will also replicate incorrect information, both when pulling from external sources such as websites, or internal sources like company PowerPoint presentations.

“Right now we’re riding the wave of having access to more information as a good thing for the models in order to come up with a great answer, but it increases the risk of false information,” Goren said.

“How many PowerPoint decks have actual factual errors in them? It turns out it's not a small number," Goren said. "Even though a human being blessed it, put it together and put it in front of their boss and they know there's a lot of risks, there are mistakes that are made through that process that are never double checked because nobody has time to go back and check every PowerPoint right after the fact.”

However, this hiccup with processing models “feels very much like a moving target,” he added, indicating these risks are going to evolve as the sophistication of both the techniques and the models grow.

“It feels like an engineering problem that will be solved,” Goren said.

Treat ChatGPT like an intern (and other tips for large language models)

Despite the technology being new, many of the risks associated with ChatGPT can be mitigated by users understanding how they need to interact with natural language processors and the limitations of these models.

First and foremost, users must understand how to format queries in order to draw the correct data from these models. Even with ChatGPT prevailing over typos and other syntax errors, it is still important to give these flexible models context within queries.

“Whenever we ask ChatGPT to answer a question, we give it the context and say here's the information that you are looking at in order to answer the question,” Goren said – a concept which is “a retrieval augmented approach.”

Users also need to confirm that the information presented to them is coming from the correct place before they take business action.

“You want to confirm that, OK, if it says this, can I see the source of where that came from? And do I trust that source so we can point you back to the source he came from?” Goren said. “If you're going to execute some business plan, it's not sufficient to just say, OK, here's the analysis and that's the end of it. It's very much understanding what was in the analysis,” he added. “It's less about even did I get the right facts? It's sort of like do the facts even matter? You need the right facts to make the right business decision.”

The narrative then becomes a centerpiece of how users would interact with any analyst, so if they ask a question, they don't just expect back a number or even a chart. In order to make the most of these models, users should keep the conversation going. The ability to respond to follow-up questions is one of the unique features of conversational AI.

“Leveraging the language model to say, ‘Oh, I understand the context of the conversation so far, plus the user's latest question, and now I can formulate a sensible response as opposed to starting from scratch,’” Goren said.

Importantly, users should keep in mind that models like ChatGPT are not ready to “tell us what to do,” he added. Instead, businesses should see these processors like an intern analyst-in-training. “You should be able to interrogate and interact with it in a way that feels very human. But it's not going to be ready to take over the job, I think, anytime soon. But I will never say never. These things are growing in intelligence pretty fast,” he said.

AnswerRocket’s Max rides ChatGPT progress

AnswerRocket recently launched Max, a conversational AI assistant that combines OpenAI’s GPT-4 model with AnswerRocket’s augmented analytics platform.

Gorem said when ChatGPT was released in late 2022, it was an inflection point for AnswerRocket because it brought the AI chat experience to the general public. AnswerRocket decided to bring the AI tool to its large customer base across the consumer goods, retail, pharmaceutical and financial services verticals.

Essentially, Max uses the GPT API to give users access to the underlying LLM, but uses AnswerRocket’s platform for specific domain context. ChatGPT alone doesn't know specifically about a problem that a brand manager is trying to solve for AnswerRocket’s customer, Anheuser-Busch, for example.

If ChatGPT is an intern, Max has already climbed higher up the ladder, Goren claimed.

“I would call GPT the intern. I would say Max already has layered domain concepts for each industry, the issues that [AnswerRocket] tackles, so it understands market share and what market shares is about as opposed to just saying, OK, there's other people sales and there’s my sales.”

Still, AnswerRocket is focused on improving Max’s capabilities.

“We definitely talk about his IQ points. We want to add IQ points every quarter to Max, but IQ probably doesn’t speak as much as industry knowledge for being maybe a senior analyst as opposed to a junior analyst," Goren added.