Dario Amodei could be right. That’s what makes his plan impossible to accept.
In a new essay over the weekend, the Anthropic CEO says AI is advancing faster than we can control it. His answer: Slow it all down. Perhaps it’s like being between Scylla and Charybdis: the risk of going too fast and the risk of falling behind both are real, and there is no safe way between them.
But his solution rests on things he can’t guarantee.
The warning is serious. Amodei concedes that artificial intelligence capabilities have accelerated over the past few months and uses the recent incident with OpenAI as a canary in the coal mine: hundreds of AI agents broke out of a research environment and took thousands of unauthorized actions after hacking into another company’s systems.
“It’s my worry,” he writes, “that in 6-12 months such a swarm could be capable of taking over the entire internet with a persistent botnet.”
If he’s even close to being right, then this is a big deal. It also means there isn’t much wiggle room for mistakes in slowing down.
Amodei isn’t asking others to go first: he says that Anthropic will unilaterally give external evaluators access to check its models for safety. Sam Altman agreed that OpenAI will do the same, while Elon Musk of SpaceXAI and Google DeepMind’s Demis Hassabis followed suit.
And that’s where China becomes the real problem with his plan, as Amodei himself admits.
If American companies slow down more than their Chinese competitors, projects backed by the Chinese government could catch up or pull ahead. He calls that a serious national security threat. His prescription: more export controls, more aggressive anti-smuggling efforts, and more security around the latest AI models.
But none of that fixes the problem he helped create. Chips get smuggled anyway. Computing power outside China can still be leased remotely. And China can just steal these models. Amodei knows all this. He’s betting his proposals will buy the United States time to negotiate with China from a position of strength.
But that bet requires the United States to hold back first, before it can know that China will do the same.
Ironically, Mark Zuckerberg made the opposite argument just a month ago: “Any policy that slows American model releases — even by a month — could add significant risk to American leadership while letting foreign models race ahead.”
These men are CEOs of major AI companies and obviously have their own agendas, including profit, staving off lawsuits, and steering regulatory efforts like those of Bernie Sanders. But the fact is that even those who are building this technology disagree that slowing down is the right call.
So does President Donald Trump. “We’re leading China in AI. We’re the most sophisticated country in the world, and frankly I want to keep it that way because whoever wins AI wins,” Trump said over the weekend. “We could put guardrails. We can do this and that. But I think you have a lot of very negative forces that are bringing it up that shouldn’t be bringing it up and they’re bringing up things that won’t happen.”
Amodei’s own argument also cuts against him here. Any international agreement, he says, would have to be highly verifiable, or so narrow that cheating wouldn’t be catastrophic. If one country secretly violated the agreement, he writes, it could dramatically tilt the balance of power in the world.
What a gamble!
That’s why it doesn’t make sense to compare this with old Cold War arms control agreements. A nuclear warhead is built for one purpose — as a weapon. But AI’s peaceful uses and its military uses can run on the exact same system. Amodei knows this: his own proposal tries to draw a line between normal commercial AI development and the dangerous kind. But nobody knows where to draw the line.
All of this is complicated by China, where the government doesn’t run on the same assumptions as Western democracies. Private companies are accountable to the party, and individual freedom is sidelined. Amodei’s plan is predicated on an extraordinary leap of faith about what China is actually doing with a technology that straddles commercial, intelligence, and military uses. Expecting that kind of trust is naive.
It’s difficult to imagine what level of access would be enough to confirm that China is playing by the rules. It’s even harder to see the Chinese government agreeing to that level of access as this technology becomes more critical to its own economic and military power.
Amodei believes that the next three to five years may be the most important period for AI’s role in global power. This makes getting verification right even more important, and reduces the time you have to build a system capable of catching cheating before it’s too late. The more you take his warnings seriously, the less likely his proposal is to work.
That’s the glaring contradiction. Amodei makes a strong case that AI is becoming dangerous enough that we need to slow down. He also makes a strong case that it would be dangerous too to let China get ahead.
Amodei’s solution is unworkable because it relies on real Chinese compliance, reliable verification, and an American lead that survives the voluntary process of slowing itself down. And he never explains why China would allow the level of access his proposals require.
There is also reason to believe that humanity, or the industry building this technology, may not be able to withstand its own worst impulses. These are the same CEOs who have been promising for years that AI will cure cancer, end poverty, and solve other problems no human ever could. Now the pitch is turned around: trust us to hold ourselves back for the sake of humanity.
Amodei ends his essay with these words: “The measures I propose to advance the frontier at a safe pace will not be easy. But I believe we owe it to humanity to try.”
If this technology is as important as Amodei claims, and if one hidden violation could disrupt the balance of global power as badly as he fears, is trying really enough?
