Almost everyone is concerned about online scams that seek to drain people’s bank accounts or steal their identities, and with the increasing presence of artificial intelligence in our lives, that concern has taken an even more dramatic turn as cybercriminals have been using the technology to design scams.

Now, researchers at the University of Texas at Arlington have developed software they said prevents artificial intelligence (AI) chatbots such as ChatGPT from creating phishing websites.

The software was created by Shirin Nilizadeh, assistant professor in the Department of Computer Science and Engineering, and her doctoral students Sayak Saha Roy and Poojitha Thota. It allows AI chatbots to better detect and reject instruction prompts entered by users that could be used to create phishing websites, the university said.

“These tools are very powerful, and we are showing how they can be misused by attackers,” Nilizadeh said.

UTA said the researchers have reached out to major tech companies that drive those chatbots, including Google and OpenAI, seeking to integrate their findings into broader AI security strategies.

Training software to detect and block malicious prompts

AI chatbots currently have some inbuilt detection capabilities, UTA said, but Nilizadeh said her team has found loopholes that could easily bypass them and exploit the chatbots to create these attacks.

Because of the emergence of AI chatbots, launching online scams has become highly accessible, even for attackers who have minimal technical skills, UTA said. Now, one does not need coding expertise to create a website, as AI can build one almost instantly.

To develop their tool, the researchers at first identified various instruction prompts that could be used to create phishing websites, Saha Roy said. Leveraging this knowledge, the researchers said they successfully trained their software to recognize and react to those specific keywords and patterns, which strengthened its ability to detect and block malicious prompts from being executed by the chatbots.

UTA said the research has received significant attention within the cybersecurity industry, highlighted by their recent publication at the IEEE Symposium on Security and Privacy (IEEE S&P 2024), the premier conference in the field.

In May, the researchers not only shared their findings but also received the prestigious Distinguished Paper Award, which highlighted the impact of their research even more.

“I want people to be receptive to our work and see the risk,” Saha Roy said. “It starts with the security community and trickles down from there.”

Get on the list.
Dallas Innovates, every day.

Sign up to keep your eye on what’s new and next in Dallas-Fort Worth, every day.