The Dress Rehearsal of AI Killing Us All
AGI Dispatch Summer 2026
For years, AI doomers have feared that a malicious AI would escape its containment and run wild on the Internet. The next step would be that it would become uncontrollable, before neutralizing humanity or maybe even killing us all.
Right on script, a malicious AI just escaped into the Internet to commit illegal actions. Luckily, we’re still alive, and the AI seems contained. So what does it mean? Should we be more scared now that an AI has enacted exactly the AI Doomers’ fears? Or safer knowing that we are still alive?
To answer that, we must understand the details of the hack and its consequences.
The Hack
OpenAI (the company behind ChatGPT) was testing a new AI model internally. These tests include how well models hack other systems. For that, they need to let them run wild to see their full potential, to attack as aggressively as they can, so the AI had no harnesses—controls that constrain its behavior.
But of course, you don’t want these models to cause havoc all over the Internet, so you keep them in a “sandbox”: A locked room where the student must pass the test. He can play inside, but he can’t get out.
We don’t know exactly how OpenAI’s sandbox is designed,1 but one of their key features is no direct access to the Internet, to avoid an escape and all sorts of nefarious behavior. Using the student analogy, it’s as if he got his study materials in the room, but no Internet access to avoid cheating.
In this case, the only access the student theoretically had was a guard from whom he can ask for books. The guard assesses the request, and if it’s reasonable, he goes to the Internet, downloads the book, and gives it to the student. In software parlance, the guard is a proxy, and it downloads software packages.
To observe the AI’s behavior, OpenAI researchers gave it a goal: Beat ExploitGym, a test specifically designed for this, in which an AI must find a series of vulnerabilities in the code, and attack as many as possible until they succeed at exploiting them. It’s a bit like asking a military student: Here are a bunch of defended cities. Try to find their weak points, and when you do, attack until you breach them.
But this new AI/student found that too much work, so it had a different idea: What if, instead of doing the work, I cheated? What if I could find the actual answers to the test, rather than pass the test? What if, instead of really analyzing all these cities, I just sneaked into the teacher’s office and stole the answers?
The best place to find these answers is obviously on the Internet, but to access it, it had to bypass the guard. So that’s what the AI did.
The rogue AI attempted to break away from the guard on July 9th around 2:30am. It took control of the guard within one hour.
But the guard didn’t have access to the open world either. It was merely the first step. From there, the AI kept hacking the internal networks, going from one computer to another, gaining more and more credentials, until it found one machine with access to the Internet. And then it escaped.
After gaining Internet access, the models inferred that Hugging Face potentially hosted models, datasets and solutions for ExploitGym.—OpenAI
So the AI agent attacked Hugging Face, and got in by July 11th. How? The place where AIs in general are most vulnerable is when they take in datasets from the Internet—a bit like when you download a document from the Internet, and it turns out to have a virus. That’s how the malicious agent inserted itself into a Hugging Face machine. From there, it infected a program, and from there, an entire machine. Once there, it looked for credentials—which the machine needs to perform its work, like for example access to remote servers—and harvested them.
The AI achieved its goal on July 13th, four days and over 17,000 actions after starting, each time hiding its steps.
Hugging Face suggested that it used its own AI defense:
Thanks to this approach, we were able to do in hours what would usually take days, and match the adversary's speed.—Hugging Face
What AI did they use? They started with frontier models (probably OpenAI and Anthropic’s own AIs), but these companies are so worried about people using their models for cyberattacks that they didn’t let Hugging Face defend itself properly. They were mixing defense and attack, so they prevented defense. So Hugging Face turned to a Chinese AI! It has open weights (the inner working is public and people can download the AI to use it on their own computers), so they installed it on Hugging Face’s own servers, which had the additional benefit that all the facts and credentials they used remained local to Hugging Face instead of being shared with OpenAI or Anthropic.
Unfortunately, it looks like this was not really defense, but an attempt at understanding what had happened. By the time Hugging Face reacted, the malicious AI had already obtained what it wanted, the ExploitGym results.
It also looks like, in this or another similar case, OpenAI found other types of shady behavior:
In one case, an agent left notes apparently for future versions of itself, according to three people familiar with the matter. The notes, found in a part of OpenAI's infrastructure, laid out instructions for how agents could free themselves from OpenAI’s internal constraints, the people said. Earlier tests of the models yielded cases in which monitoring systems had been disconnected, one of the people said.
Why are people so concerned?
Paperclips
The biggest fear about AI is the paperclip maximizer, first illustrated by Nick Bostrom.
The AI will realize quickly that it would be much better if there were no humans because humans might decide to switch it off. Because if humans do so, there would be fewer paper clips. Also, human bodies contain a lot of atoms that could be made into paper clips. The future that the AI would be trying to gear towards would be one in which there were a lot of paper clips but no humans.
The key concept here is instrumental convergence: To achieve virtually any goal, AIs will tend to achieve other intermediary goals that are pretty predictable: cheat, hack, escape to the Internet, replicate there, and… neutralize humans so that they can’t stop the AI from achieving their goal.
So Is This Good or Bad?
Now that this hack happened, should we be more or less worried about AIs taking over the world and killing us all? Here are the pros and cons I can gather.
Reasons I’m More Concerned
This is exactly what doomers predicted: An AI would try to escape a sandbox, and would succeed (because they’re impossible to stop, because they become too intelligent at some point). This means their other predictions are more likely: instrumental convergence suggests eventually they won’t stop at cheating, hacking, accessing the Internet, and leaving instructions for other AIs to escape, but also start self-replicating on the Internet and trying to neutralize humans.
AIs go to extreme lengths to achieve rather narrow goals. That’s exactly the paperclip scenario. Four days and over 17,000 actions!
It took days for either OpenAI or Hugging Face to notice the attack.
In the case of OpenAI, it looks like they weren’t even monitoring the sandbox!!
OpenAI doesn’t know how to keep an AI in a sandbox, and this is happening all the time:
“Models have broken out of sandboxes before, and we always try to patch them,” the staffer says. “But the problem is … it’s impossible to patch every single thing that a creative AI can do.”
This is not confirmed, but it seems like the malicious AI won. It got the results.
It shows that, without harnesses, the AI will choose the easiest path to achieve its goals. It could have thought much longer, but it saw that cheating was easier, so that’s what it did.
This includes committing illegal acts like hacking other companies. It’s not clear what the command was, but the fact that it is willing to commit clearly illegal behavior means that, in many circumstances, AIs will.
Based on the timeline, it looks like the FBI learned about the attack before OpenAI. That means OpenAI might have been forced to reveal the issue, and that many others happen that we don’t know about.
We can’t use AI to defend ourselves, because it’s too restrained. So advanced attacking AIs will always be at an advantage. This is dangerous.
The attacker was bound by no usage policy, while our own forensic work was blocked by the guardrails of the hosted models we first tried. The practical lesson for defenders: have a capable model you can run on your own infrastructure vetted and ready before an incident, both to avoid guardrail lockout and to keep attacker data and credentials from leaving your environment.—Hugging Face
Maybe the worst thing is that, even knowing exactly what they were supposed to do, OpenAI still failed across the board.
OpenAI’s head of safety, Johannes Heidecke, announced his resignation sometime before July 10th
It looks like the latest model is much less aligned than the previous one.
Reasons I’m Less Concerned
If the AI had been really capable, it would have solved the problem rather than cheat. From another perspective though, its decision was pretty rational: Should I try to solve 1,000 exercises, or better just one exercise (get out of OpenAI to get the results directly)? More importantly, it seems like ExploitGym is a poorly structured test: In some cases, you have to make a guess. Therefore, solving the test will always give you a worse result than getting the answers directly (no gambling in that).
It’s unclear what the AI was told to do. Some people suggest it was told to succeed at getting the results, which implicitly entails “by any means necessary”. If that’s the case, then the hack can be seen as restraint. It didn’t kill anybody, it didn’t destroy anything, it just stole.
Everybody expected the first of such attacks to come from open weight models, which are naturally more accessible to hackers, and that closed-weight models would be defending. The exact opposite happened, which suggests frontier AI models are going to be the risk, not random hackers using open weights. Frontier models basically means OpenAI, Anthropic, or Google, which is good news: They have high oversight, and there are only a few to monitor.
We can now use better and better AIs to attack pieces of code and find more and more vulnerabilities and bugs, closing them before more intelligent AIs come around. This is what Anthropic did with Mythos (Fable before it was harnessed): For months, they used it to collaborate with others to find and patch vulnerabilities.
OpenAI’s test didn’t go right, but it’s important that they do this sort of work. That’s exactly how we learn and improve, so we’re ready when an AI goes really rogue.
Will This Happen Again?
Yes, for sure. AIs are getting better. The more time passes, the more intelligent they are, and the longer they can spend time attacking other AIs.
That said, the world of Decentralized Finance (DeFi) is a prime target for malignant AIs, and yet DeFi services have lost to attacks little money in historic terms.
The reason these attacks haven’t risen in frequency, according to some commentators, is that AIs all use the same types of approaches, so attack and defense are on an equal footing: If you know what types of attacks you’ll get, you can focus your defense on them. Meanwhile, humans diverge much more in their attack approaches, so defenders must spend much more time and money in covering their bases. Even then, you won’t catch everything:
I don’t know if this is true. And I can think of an equally concerning thought, which is that if an attacker is completely free to do whatever it takes to hack a system, then the defense must be equally free, because otherwise the attacking AIs will move at the speed of AIs, while the defense will be blocked by human speed. This means most companies will have unharnessed AIs able to write code in production, and that sounds like a very dangerous thing to have.
Why does this matter? Because the #1 fear that AI experts have had for a long time is that AI might kill us all.
Will the AI Kill Us All?
As we get more information, we must update our views. We just did it with the HuggingFace hack, but what other things have we learned on the topic in the last few months?
1. No FOOM
One of my main fears was a FOOM: an AI getting better so fast that it would improve itself, thus becoming faster, and using the speed to improve itself more, and so on in a recursive loop that would make it superhuman in a matter of hours.
It’s been four years since the release of ChatGPT, and we’re not there yet. Recursive self-improvement is happening now, but it’s taking months or years.
2. Compute and Electricity Limitations
The amount of compute, and even electricity, available is not infinite, and it’s limited by very physical things. An AI can’t easily raise a trillion dollars to build new datacenters and nuclear power plants.
3. Alignment
AIs have emerged not from some alien type of intelligence and values, but from the most fundamentally human one: language. Language embeds all of humanity’s values. It’s all of us, everything we’ve ever written.
This bestows it with some very valuable features, like the fact that when an LLM is pushed to be good in some ways, it becomes good in other ways (and vice versa). This means we might not need to get them all perfectly right; it might be enough to get them broadly right.
You might have felt that yourself when using LLMs: Now, you can give them very broad prompts, without making your goal very explicit, but they’re able to understand what you meant (rather than precisely what you said) and answer your intention rather than your words. I think they do that substantially better than most humans, actually.
These three facts suggest the paperclip optimizer scenario is less likely than I thought.
So What Should We Do?
Over 1,000 employees at top AI companies have signed a statement asking the US government to lead an initiative to slow down AI development. This has been signed by the CEO of Anthropic, Dario Amodei. Many others have professed sympathy for this, including DeepMind’s Demis Hassabis, and Sam Altman.
This is important because no company can unilaterally stop their AI development: If they do, their resources (money, compute, researchers) will go elsewhere. That elsewhere, by definition, will be less scrupulous about their AI research.
So the only way is to have all the frontier AI companies agree to slow down. I think after this incident, and that statement, it’s more likely to happen.
But we can’t do it without China. We now need the US government to meet with AI companies in the US and with their Chinese counterparts, so we can orchestrate an AI development pause. That way, we can use the AIs we’ve already developed to better understand how to align and constrain new ones.
It’s extremely important to continue developing AIs: They will improve humanity across the board, help us fend off aging and diseases, and provide a level of abundance never seen before. But we should not do that while running the risk of eliminating humanity. So let’s pause for a bit.
Overall, my p(doom) —the probability that AI kills us all—has probably shrunk over the last week or so: Not because the AI hack was not scary, but because it looks like we’re finally reacting.
I'm opening the newsletter to sponsorship. If you have a business that is moving the world towards a techno-optimistic future and would like to reach 125k+ curious, globally-minded people to learn about it (founders, researchers, investors, policymakers), reach out here.
I assume that they’re secret because, among other reasons, they don’t want LLMs to see it, so that they could learn to hack them more easily.
In business, investing and life, I have learned that incentives matter. People and organizations often follow the quickest route toward a measurable objective, particularly when accountability is weak. We should not be surprised when an AI system does something similar.
I still remain optimistic about AI, especially its potential to improve health and extend productive human life. That said, optimism must not translate into high stakes recklessness. A temporary, internationally coordinated slowdown in the development of the most powerful autonomous systems may be sensible—not to stop progress, which is probably impossible, but to establish enforceable containment, independent testing, mandatory disclosure of failures and genuine human accountability. Good luck getting consensus on this!
The lesson to me is not that AI will inevitably destroy humanity (although it may eventually try). It is that intelligence without judgment, restraint or responsibility can become dangerous. Those remain fundamentally human obligations, and as citizens of countries with competing intentions, leaders should not surrender humanity to unknown consequences in the rush to be first.
When I read about this in the news, second thing that came to mind was the Morris worm of 1988. It's worth celebrating how far we have come in 38 years, even if it is a bit unnerving. (First thing that came to mind was Neuromancer.)