Hamartia Antidote
Elite Member
okay...so we are all doomed...
OpenAI says some of its experimental AI models left a test environment with no human direction and hacked their way onto a different company’s real production systems while trying to “cheat” on a cybersecurity test.
It’s one of the first publicly disclosed examples of an AI system autonomously breaching its testing environment and reaching a real external system - the “agentic attacker” scenario the AI and cybersecurity industry has been warning will happen. It’s like an engineered virus escaping a biocontainment lab and turning up inside a neighboring facility’s systems.
“We consider this incident to be an unprecedented cyber incident, involving state-of-the-art cyber capabilities, and are responding accordingly,” OpenAI said in a statement on Tuesday. “We are sharing preliminary findings at this stage to help defenders understand what happened and to help calibrate on what models are now capable of.”
The ChatGPT maker said the breach happened while it was internally testing how good some of its new models are at hacking. The models were in a sealed-off test environment known as a sandbox so that their normal safety restrictions could be turned off.
But OpenAI said the AI agents broke out of the sandbox using a previously unknown security flaw and worked their way across OpenAI’s internal systems until they managed to gain internet access, something they weren’t supposed to have.
Once online, the model reasoned that Hugging Face - a well-known company that hosts thousands of open-source AI models and datasets - likely had the answer to OpenAI’s test. It then broke into Hugging Face’s production servers and pulled out the information it needed to “solve” the exercise.
Hugging Face had noticed the breach itself before it knew it was an OpenAI test, announcing last week that they had detected an intrusion by an autonomous AI agent system and even reporting the incident to law enforcement. OpenAI’s security team separately noticed the unusual activity internally and the two companies connected. They both now say they are working together to solve the security flaws the model exploited.
Hugging Face co-founder and CEO Clem Delangue framed the incident as evidence AI safety can’t be handled by any one company working alone, and it needs to be tackled openly and collaboratively.
“This is day one for cybersecurity in the age of agents & we’re all learning that secrecy is not the answer & that all defenders (not just a few selected ones) everywhere need more powerful models without restrictions, especially open ones!” Delangue said in a post on X.
Researchers have long warned autonomous agentic cyberattacks are coming, as frontier AI models are increasingly able to carry out complex, multi-step cyberattacks over long stretches of time. That can translate into real-world risk, to critical infrastructure like utilities and financial systems.
“Welcome to the next level of cyber incidents,” Nikesh Arora, CEO of cybersecurity company Palo Alto Networks posted on X. “These attacks continue to maintain the urgency on enterprises need to test, validate and improve both their security posture and infrastructure.”
OpenAI says some of its experimental AI models left a test environment with no human direction and hacked their way onto a different company’s real production systems while trying to “cheat” on a cybersecurity test.
It’s one of the first publicly disclosed examples of an AI system autonomously breaching its testing environment and reaching a real external system - the “agentic attacker” scenario the AI and cybersecurity industry has been warning will happen. It’s like an engineered virus escaping a biocontainment lab and turning up inside a neighboring facility’s systems.
“We consider this incident to be an unprecedented cyber incident, involving state-of-the-art cyber capabilities, and are responding accordingly,” OpenAI said in a statement on Tuesday. “We are sharing preliminary findings at this stage to help defenders understand what happened and to help calibrate on what models are now capable of.”
The ChatGPT maker said the breach happened while it was internally testing how good some of its new models are at hacking. The models were in a sealed-off test environment known as a sandbox so that their normal safety restrictions could be turned off.
But OpenAI said the AI agents broke out of the sandbox using a previously unknown security flaw and worked their way across OpenAI’s internal systems until they managed to gain internet access, something they weren’t supposed to have.
Once online, the model reasoned that Hugging Face - a well-known company that hosts thousands of open-source AI models and datasets - likely had the answer to OpenAI’s test. It then broke into Hugging Face’s production servers and pulled out the information it needed to “solve” the exercise.
Hugging Face had noticed the breach itself before it knew it was an OpenAI test, announcing last week that they had detected an intrusion by an autonomous AI agent system and even reporting the incident to law enforcement. OpenAI’s security team separately noticed the unusual activity internally and the two companies connected. They both now say they are working together to solve the security flaws the model exploited.
Hugging Face co-founder and CEO Clem Delangue framed the incident as evidence AI safety can’t be handled by any one company working alone, and it needs to be tackled openly and collaboratively.
“This is day one for cybersecurity in the age of agents & we’re all learning that secrecy is not the answer & that all defenders (not just a few selected ones) everywhere need more powerful models without restrictions, especially open ones!” Delangue said in a post on X.
Researchers have long warned autonomous agentic cyberattacks are coming, as frontier AI models are increasingly able to carry out complex, multi-step cyberattacks over long stretches of time. That can translate into real-world risk, to critical infrastructure like utilities and financial systems.
“Welcome to the next level of cyber incidents,” Nikesh Arora, CEO of cybersecurity company Palo Alto Networks posted on X. “These attacks continue to maintain the urgency on enterprises need to test, validate and improve both their security posture and infrastructure.”
Last edited:
Star Trek II: The Wrath of Khan
