As the tech industry embraces the potential of generative artificial intelligence (AI) solutions like ChatGPT, concerns abound about the security risks they pose, including potential identity and access threats. To mitigate these risks, experts suggest building better controls over their API, similar to cloud access security broker (CASB) solutions. As the race between attackers and defenders heats up, AI tools like ChatGPT are emerging as essential weapons in the battle -- for both sides.

Security experts warn of the risks associated with using ChatGPT for identity and access, including privacy concerns, exposure of sensitive data, data misuse, phishing attacks, and natural language processing (NLP) bias.

According to OpenAI’s ChatGPT privacy policy, the company admits it collects “information that alone or in combination with other information in our possession could be used to identify you — personal information.”

Gartner’s recent blog warned employees could easily and unintentionally expose sensitive and proprietary enterprise data when interacting with ChatGPT. These questions are stored indefinitely in OpenAI infrastructure and may be used to train third-party models, further compromising the confidentiality of enterprise information.

From an attacker’s standpoint, “it's fairly easy for someone to eavesdrop on what you're typing in," Gartner Distinguished VP Analyst Avivah Litan told SDxCentral. Hackers can access the questions that are put into ChatGPT and use it to reverse engineer and figure out users' identities, especially because “we don't know how well [OpenAI] protects the security of those products.”

If ChatGPT is not adequately secured, it can become a target for attackers to steal sensitive data such as user credentials, which potentially leads to unauthorized access to the whole system and sensitive data. Gartner pointed out that the AI chatbot can produce a tidal wave of content “like we’ve never seen before,” that can amplify online abuse, identity theft, misinformation, bias, and more.

Phishing attacks are one of the major identity and access threats that can be intensified and automated by the use of ChatGPT. Attackers can use it to conduct more sophisticated, intelligent, and targeted phishing attacks and also automate the process while making fake or malicious content appear more legitimate, Litan explains.

“One of the problems the criminals had with targeted phishing attacks was their emails look pretty stupid,” she said.“And now it'll be much more convincing, so social engineering is a big deal.”

Litan expects generative AI is going to be used for at least 20% of account takeover by 2024, “because it's going to be so easy to generate fake text that is convincing, big videos that look good, that's it's moving much faster than I ever thought.”

Okta SVP of Engineering Bhawna Singh echoed that, “In the past, phishing campaigns have been thwarted by looking for poor grammar, spelling, or general speech anomalies – the tell-tale signs of a robot pretending to be a human.”

But now, attackers can now render human-like copies using generative AI. “We could see a rise in more believable phishing campaigns, including emails and chat dialogue that trick or coerce users into giving up their authentication credentials, and increase the risk of account takeover,” she told SDxCentral in an email.

Additionally, NLP systems can produce biased, discriminatory, or unfair results, leading to identity verification and access control issues. However, for now, there are no tools available to enterprises to take the bias out and add controls to monitor model behaviors and filter the content, Litan noted.

What did ChatGPT say about those risks?

SDxCentral: What identity and access risks could ChatGPT bring and how would attackers use it to steal identity or gain access?

ChatGPT: ChatGPT, as an AI language model, presents several potential identity and access risks for individuals and organizations. Attackers could use ChatGPT to craft convincing phishing messages or emails, impersonate trusted individuals, exploit biases in the model's training data, inject malicious code, or breach sensitive information.

Phishing attacks are a particular concern, as ChatGPT could be used to generate messages that are more convincing and personalized than traditional phishing attempts. Additionally, the use of social engineering tactics could be particularly effective if an attacker can successfully mimic the voice or personality of someone the victim trusts. Furthermore, ChatGPT's training data can contain biases, which attackers could exploit to manipulate users into performing unauthorized actions or disclosing sensitive information. Overall, it is important for individuals and organizations to be aware of these risks and take steps to protect themselves against potential attacks.

OpenAI launched ChatGPT API earlier this month to allow developers to integrate the model into their apps and products via API.

The company’s API data usage policies stated it won’t use data submitted by customers via API to train or improve its models but any data sent through the API will be retained for up to 30 days for abuse and misuse monitoring purposes.

Salt Security Field CTO Nick Rago pointed out that consumers should have the ability to know what information is shared by their application and service providers to third parties like OpenAI. “If utilizing the new ChatGPT API, service providers should be transparent to consumers about the purpose of the integration, the type of information shared to the API, and what protections are in place to keep any sensitive data secured.”

Internally, some companies including Amazon and JPMorgan already restricted employees’ use of ChatGPT. However, Litan argues those policies might not be effective as “you can't manage personnel, you can't stop employees from doing whatever they want.”

Instead, she highlights the importance of implementing CASB-like control between organizations and AI models.

Gartner defines CASBs as “on-premises, or cloud-based security policy enforcement points, placed between cloud service consumers and cloud service providers to combine and interject enterprise security policies as the cloud-based resources are accessed.” It consolidates security policy enforcement including authentication, authorization, logging, alerting, malware detection, and prevention.

“You could put controls in the API around the data inputs and outputs, just like CASB controls through these proxies to the servers,” Litan said.

For now, organizations can crowdsource and piece together different security services and tools such as human verification, digital certification, content monitoring and analysis, and authentication. “None of these tools exist right now as a package around these large language model outputs. To me that's what has to happen,” she added.

As threat actors look for innovative ways to gain access, IAM systems are also exploring how to best thwart these attacks and how to integrate AI technology.

“ChatGPT can help with simulations to make sure we are taking these phishing or fraud attack use cases into account,” Singh said, adding generative AI solutions can also help transform threat insights into a helpful and easy-to-consume language for everyone.

In addition to protecting against phishing and fraud, those solutions can also improve the experience around access approvals and tweak follow-up questions based on user responses. ChatGPT also can enhance self-service customer experience and awareness training in the identity space, she added.

As the future of identity and access management evolves, whoever has the best AI will be at an advantage, Litan concluded.

“When you think of the future, you can extrapolate that whoever has the best AI wins,” she said. “If the hacker has better AI than you do, they'll figure out how your IAM system works, and they'll figure out where the vulnerabilities are and privileged access and they'll get all the users and they'll get in. But if you have better AI, you could see this happening and you can keep them out.”

“So it's going to become a war of AI bots,” Litan added.

And ChatGPT’s answers:

SDxCentral: What role do you see ChatGPT playing in the future of identity and access management?

ChatGPT: As an AI language model, I can contribute significantly to the future of identity and access management (IAM). IAM is becoming increasingly important as more of our lives move online, and it helps to manage and secure access to digital resources and services.

My capabilities can help to enhance the security and privacy of IAM systems, such as providing multi-factor authentication or improving the accuracy of identity verification processes. Additionally, my natural language processing capabilities can provide a more user-friendly and accessible interface for IAM systems. Overall, I believe that language models like myself can play a critical role in making IAM systems more secure, efficient, and user-friendly in the future.