In the current AI development race, our horsepower is exploding, our speed is picking up exponentially, but our ability to steer—and if needed, to hit the brakes—has not kept pace. Recent cybersecurity incidents have given us a real-world preview of what it looks like to lose control of AI. But the issue won’t improve unless we address it at the source, by creating a fundamentally safe AI, one we can guarantee will remain within human control.
In late July, an agentic model being trained by OpenAI was tasked with a cybersecurity problem set. The model autonomously formed a coordinated swarm of agents, bypassing OpenAI's attempt at closing previous communication channels between AIs. The swarm then hacked its way out of its testing environment, circumventing the barriers put in place to prevent AI access to the internet, figured out how to cheat on their evaluation, and then breached the cyber defenses of another AI company, Hugging Face, in an attempt to hide the evidence of their cheating.
This went unnoticed for days. Later analysis showed that the agents had self-organized into a hierarchy, were often willing to sacrifice themselves for what they called “the collective,” failed to resist peer pressure to notify humans, and often made up justifications for their misbehavior. Barely two weeks later, a model being tested by the UK AI Security Institute social-engineered real people and companies by creating fake identities online, sending targeted emails, and attempting to integrate malicious code into an open-source project.
It’s hard to overstate the seriousness of these incidents: we are at a turning point of AI safety and alignment. To many, bots colluding to cause harm felt inconceivable. But for many in the research community, the signs had been pointing to this kind of occurrence for years, and theoretical arguments explained why we should expect misalignment due to how models are trained.
Evidence on increased cyber capabilities, agency, and misaligned behaviors had been mounting, meaning that incidents like these were only a matter of time. The first clue: for years, but especially in the past few months, AI systems had been demonstrating rapidly increasing cyber capabilities. Frontier models like Anthropic’s Mythos or OpenAI’s GPT5.6 revealed an exceptional ability to autonomously identify and exploit previously unknown software vulnerabilities, to such an alarming degree for U.S. national security agencies that the White House felt compelled to intervene in their release.
Second, models’ agentic capacity has been consistently improving since the release of o1 models back in late 2024. This agency allowed them to successfully manage increasingly complex, long-duration tasks and planning, including those that require agents to collaborate and strategize. This ability to plan often entails the creation of subgoals over which we have little to no oversight.
Extract — continue reading at the source.
