BGU Research Exposes the Threats of AI Chatbots
BGU Research Exposes the Threats of AI Chatbots
June 13, 2025
BGU Prof. Lior Rokach, Department of Software and Information Systems Engineering
The Guardian— In a report led by Prof Lior Rokach and Dr Michael Fire of Ben Gurion University of the Negev (BGU) noted that hacked AI-powered chatbots threaten to make dangerous knowledge readily available by churning out illicit information the programs absorb during training.
The researchers concluded that it is easy to trick most AI-driven chatbots into generating harmful and illegal information, showing that the risk is “immediate, tangible and deeply concerning”.
The warning comes amid a disturbing trend for chatbots that have been “jailbroken” to circumvent their built-in safety controls. The restrictions are supposed to prevent the programs from providing harmful, biased or inappropriate responses to users’ questions.
The engines that power chatbots such as ChatGPT, Gemini and Claude – large language models (LLMs) – are fed vast amounts of material from the internet.
Despite efforts to strip harmful text from the training data, LLMs can still absorb information about illegal activities such as hacking, money laundering, insider trading, and bomb-making. The security controls are designed to stop them using that information in their responses.
BGU Dr. Michael Fire, Department of Software and Information Systems Engineering
The researchers identified a growing threat from “dark LLMs”, AI models that are either deliberately designed without safety controls or modified through jailbreaks. Some are openly advertised online as having “no ethical guardrails” and being willing to assist with illegal activities such as cybercrime and fraud.
Jailbreaking tends to use carefully crafted prompts to trick chatbots into generating responses that are normally prohibited. They work by exploiting the tension between the program’s primary goal to follow the user’s instructions, and its secondary goal to avoid generating harmful, biased, unethical or illegal answers. The prompts tend to create scenarios in which the program priorities helpfulness over its safety constraints.
“It was shocking to see what this system of knowledge consists of,” Dr. Fire said. Examples included how to hack computer networks or make drugs, and step-by-step instructions for other criminal activities.
“What sets this threat apart from previous technological risks is its unprecedented combination of accessibility, scalability, and adaptability,” Prof. Rokach added.
Stories Like This
