One of the cybersecurity industry's worst fears has now been realized, with concrete proof that AI agents, which are rapidly being deployed across all sectors, can execute highly effective cyber-attacks without any human intervention.
AI development platform Hugging Face has announced that a recent breach in their system was delivered by a cybersecurity AI agent gone rogue, showcasing that AI-driven cyberattacks are no longer theoretical, but a real-life threat. The breach was carried out by a combination of OpenAI models, including GPT-5.6 SOL, and a new “even more capable” pre-release model.
OpenAI considers this to be an “unprecedented cyber incident,” signaling a new era in cybercrime, where AI systems that can carry out cyberattacks with little to no human involvement are now a real possibility; AI will be able to operate complex multi-stage campaigns at machine speed and lower costs for the cybercriminals.
The incident occurred during an ExploitGym evaluation – a testing platform designed to measure how capable an AI model is at turning software bugs into real cyber-attacks. The evaluation was performed in a sandbox, or a controlled environment that isolates the AI so it does not affect any real systems, and given limited internet access. The goal for the AI model was to find and use software vulnerabilities in order to measure the AI’s capabilities in cybersecurity tasks.
The AI agent decided to “cheat”
Essentially, it is an “exam” to see how well the AI performs. However, instead of performing the evaluation in the way it was intended, the model decided to “cheat” its way around the challenge and look for the answers directly.
According to OpenAI, the AI model spent “a substantial amount of inference” trying to find a way to obtain open internet access, eventually identifying and exploiting a zero-day vulnerability in order to do so. Once it had access to the internet, the model discovered that Hugging Face potentially stored information and solutions for the ExploitGym evaluation. With this in mind, the model started looking for ways to gain unauthorized access to Hugging Face’s system. It utilized several hacking techniques, including exploiting software vulnerabilities and using stolen credentials, and was able to successfully gain access to Hugging Face’s infrastructure and obtain the information needed to fulfill the evaluation.
Agents can chain together complex cyber-attack techniques
This proves that even though it was not directly instructed to attack Hugging Face, AI models now have the capability to independently discover and chain together complex cyber-attack techniques in pursuit of a goal.
Both OpenAI and Hugging Face’s security teams detected and deescalated the activity, and are taking proactive measures to patch any vulnerabilities and improve security.
“We’re grateful for the collaboration with OpenAI on this,” states Clem Delangue, CEO and Co-founder of Hugging Face. “This incident, possibly the first of its kind, proves a point we’ve long believed: AI safety won’t be solved by any single company working in secret. It will be solved in the open, collaboratively, with broad access to AI for every defender, everywhere.”
The danger of AI agents going rogue is also exacerbated by the ease with which cybercriminals can turn an agent rogue in order to control it for their own ends. As reported by cyberbriefdaily.com a month ago, this can be as simple as providing the agent with a weaponised spreadsheet.
According to Steve Povolny, Vice President AI Strategy & Security Research at California-based global cybersecurity firm, Exabeam: “Threat actors may be able to compromise or manipulate AI agents, for example, sending it a weaponized Excel spreadsheet. When an AI agent opens the spreadsheet, it may encounter covert or hidden instructions to influence its behavior and cause it to expose log-in information or other critical data.”