If ever there’s been a moment for a technology to grab the spotlight, this is it for generative AI.

From ChatGPT to Bing Chat to DALL-E, numerous tools continue to emerge in rapid succession.

And as they evolve, so, too, do cybersecurity risks — both when attackers leverage them to accelerate and enhance their exploits and when companies quickly and carelessly deploy generative AI (genAI) platforms.

With use only accelerating — the genAI market is expected to reach $51.8 billion in 2028, up from $11.3 billion in 2023 — organizations are fighting an increasingly contentious battle.

“This [kind of technology] is very much available to all at the same time,” said Sean Guillory, senior robotics process automation bot developer at management consulting services company Booz Allen. “It's [available] to everybody with all sorts of intentions and aspirations, some more maleficent than others.”

Evolving deception, quicker results

One particular area of concern, according to Chris Anley, scientist with IT security company NCC Group, is deepfakes, or audio or video that has been digitally manipulated to impersonate a specific person. Attackers don't need much to create one that is usable and believable.

He pointed out that invoice fraud in general is a serious problem — this is when attackers pretend to be an executive or business owner and successfully convince a finance department to pay out a fraudulent invoice. Adding genAI to the mix only makes attacks more convincing; platforms can quickly generate fake text, voice, image and video.

“The ability to impersonate a style or even an individual is one of the things we're going to have to brace for,” Anley said.

Guillory agreed, saying that rudimentary deepfakes were caught quickly, but they are getting better all the time. “We don't want to have the hubris that we've caught it all,” he said.

There are myriad other ways attackers are beginning to use genAI: identifying vulnerabilities in APIs, overwhelming (and thus sabotaging) machine learning embedded in security tools, producing fake documents, guessing passwords and beating CAPTCHA, and carrying out phishing or social engineering attacks.

Traditional phishing emails have come to be relatively easily recognizable due to rampant spelling or grammar errors or lack of cultural context, Anley noted. Now, though, an attacker can quickly and easily generate a phishing or person-specific spear phishing email that seems totally legitimate.

Furthermore, some experts have indicated that attackers are using genAI to write malware, but Anley said that risk may be overstated.

“Sure, you can use things like ChatGPT or GitHub Copilot to generate code, and that code can be malicious or can be used for malicious purposes,” he said. “But I don't think that's really all that helpful to an attacker. There’s more than enough malware out there that can be adapted and modified.”

In the end, the personalization piece is most troubling, he said.

“The real danger is in generating content that imitates specific people or targets specific people,” said Anley.

Organizations unwittingly exposing their data

Some enterprise developers have been discovered posting proprietary code and other data to third-party genAI systems. Samsung, for instance, recently uncovered three separate instances of staff misuse of the platform that leaked equipment measurement, yield data, source code and other sensitive information.

The company has since gone so far as to ban the use of ChatGPT. Verizon executives have also said that ChatGPT is not accessible from its corporate systems, and several financial and healthcare institutions also prohibit its use.

“The world is changing; folks are still figuring out how to use these systems safely,” Anley said.

Taking things a step further, attackers are targeting AI and ML systems themselves. OpenAI confirmed in March that it temporarily took ChatGPT offline due to a bug in an open-source library. Most notably, this exposed payment-related information — names, emails and physical addresses, the last four digits of credit card numbers and expiration dates — of a small percentage of paid subscribers.

“Everyone at OpenAI is committed to protecting our users’ privacy and keeping their data safe,” the company said in a blog post. “It’s a responsibility we take incredibly seriously. Unfortunately, this week we fell short of that commitment, and of our users’ expectations.”

As with everything, AI vigilance is critical

Some organizations are beginning to experiment with using AI to fight AI, but Anley emphasized that it comes down to basic security precautions and cybersecurity awareness.

Organizations should have robust identity-verification mechanisms in place along with standard DevSecOps processes, regular external security audits, multi-factor authentication, endpoint device management and VPN lockdown, he advised.

“Wherever a system can be locked down, make sure that it is locked down,” he said.

He pointed out that many organizations have a “hard outer shell,” but very few security barriers inside the data science area. Even if there are multiple barriers to entry, if an attacker compromises an individual data scientist’s laptop, they may be pre-authenticated through multiple channels.

Organizations must ensure they have a trusted internal area and empower security teams to respond. It’s also critical to have visibility into just what staff are up to, to make sure sensitive information isn't leaking, Anley said.

“All these basic security precautions help,” he said. “But the thing that makes the most difference is vigilance.”

How does the human brain interact with AI?

At the same time, it’s important to address the larger, more philosophical questions around AI, Guillory posits.

We must recognize and acknowledge that even experts can be taken by surprise, he said. Why do these systems keep fooling people? What qualities make AI feel real or not? How do pixelation and luminance play into whether we can spot AI?

These types of questions could be answered by looking at cognitive biases and analysis through neuroscience or psychophysics (or the interaction of physical stimuli and sensory systems). Insights in those areas could then bolster methods to more optimally detect and fight malicious AI.

In addressing security threats, Guillory pointed to the emerging methodology of integrating the DISARM framework and MITRE Attack into cognitive security operations centers (SOCs), or organized teams that use forensics auditing and reporting to identify, detect and thwart cyberattacks. Both DISARM and MITRE are based on the principle of sharing knowledge of adversary tactics and best practices of defense.

“That kind of know-how would be absolutely helpful,” Guillory said. “If you can stop folks once you know a certain aspect of the killchain, you can stop penetration.”

But it isn't as easy as cutting and pasting what's out there for killchain models, he emphasized. A type of governance, risk and compliance program (GRC) can help provide more of a playbook for cybersecurity leaders and teams.

Still, he said, “new things are happening all the time. We certainly are still in that discovery phase.”

Be realistic about AI

Looking at AI more broadly — beyond security — organizations must keep their expectations realistic, Anley noted.

“These systems are surprisingly capable,” he said, “but it’s important not to get caught up in the hype.”

AI systems are not really thinking. They are simply statistical systems that are correlating their training data with a prompt, he said. Also, they fail and there are many things they can’t do. For instance, some have shown trouble with basic arithmetic, when, on the other hand, they can write haikus or summarize “Jane Eyre" in different languages or without using the letter “e.”

Then there’s the “hallucination” problem. Anley said that isn’t the best word to describe it, because it’s anthropomorphizing. The better way to put it: “They lie. The content they generate will be misleading, but it will be very convincing.”

Ultimately, as he put it, if you take everything that has ever been written by any human anywhere, “it’s going to generate some amazing stuff.”

“It's been trained on us, on all of us, for centuries,” he said. “Of course it's great, we're great. We’re seeing our own content reflected back at us.”