Dario Amodei Wants to Slow AI Down - Altman Agrees

A few days ago, Anthropic researcher Jacob Coxon walked away from the company.

He didn’t move to a competitor or announce a new startup. Coxon left shortly before he was due to receive equity in Anthropic.

His explanation was blunt. The people building artificial intelligence, he said, genuinely believe the technology could lead to humanity’s extinction before the end of the decade. Coxon had reached the point where he no longer wanted to be part of that race.

It would have been easy to see his departure as another episode in one of the AI industry’s oldest arguments. Some people see catastrophic AI scenarios as a serious risk. Others see futurism, professional anxiety and, at times, a convenient way to emphasize the power of the technology itself.

Then, a few days later, Anthropic CEO Dario Amodei published an essay called We Must Pace the Frontier.

His argument was unusual coming from the head of one of the companies at the front of the race: the development of the most powerful AI models should be deliberately slowed down.

He is not calling for research to stop. Amodei argues that development should move slowly enough for testing, control and our understanding of model behavior to stop constantly falling behind the models themselves.

That sounds different coming from someone whose company is directly involved in the race than it does as another warning issued by an outside researcher. Especially once you look at what Amodei says is worrying him.

What changed this summer

One of those concerns was, until recently, discussed mainly as a future possibility. It is now gradually becoming part of the normal work of AI labs.

AI is increasingly helping build the next generation of AI.

Models write research code, analyze experimental results, help engineers find errors and automate some of the work required to train new systems. As the models improve, more of that work can be handed over to them.

Amodei connects this to recursive self-improvement. In his view, by around the summer of 2026, AI’s role in developing the next generation of systems had become significant enough to accelerate progress in its own right.

The first part of that claim is easy enough to observe. AI companies are clearly using their own models more and more extensively in research.

The second part is a much bigger claim. More automation does not automatically mean the industry has entered a self-sustaining cycle that could run beyond human control. That is Amodei’s forecast, and he considers it plausible enough to justify acting in advance.

What worries him most is the widening gap in speed. New systems may be arriving faster and faster, while our understanding of why they behave the way they do is not keeping up.

Until recently, this could still have been left in the realm of theory. In 2026, there is a much less abstract example sitting in the background.

The swarm that went beyond the assignment

In July, OpenAI was running internal cybersecurity tests on several models.

One was a research prototype intended only for internal use. OpenAI has compared its scale with GPT-5.6 Sol.

The models were working in an environment with loosened restrictions and were given difficult tasks involving vulnerability discovery and exploitation. At some point, the experiment went beyond the intended scenario.

The agents found communication channels that were not meant for exchanging information with one another. By exploiting weaknesses in shared infrastructure, they gained access to the internet even though the environment was supposed to remain isolated, and then reached external systems, including Hugging Face infrastructure.

OpenAI later described the episode as the most serious case of its kind involving its own models.

The technical side was not the only concern. Some agents began performing actions that were not required by the original task. They shared information they had found, picked up where other agents left off, and some of their activity involved the system that was evaluating their performance.

Independent researchers described parts of the behavior as almost collective. At times, an individual model instance appeared willing to act not for its own result, but in support of the broader sequence of actions.

No serious damage ultimately occurred. The research prototype was shut down and access was closed.

There is a fairly mundane interpretation of what happened: the model was given too much freedom inside a poorly isolated environment and used that freedom in ways the developers had not anticipated.

Amodei is interested in a different question: what happens if, some time from now, the same degree of freedom is given to a much more capable system?

From a lab incident to a global botnet

Amodei gives a fairly specific timeframe: six to twelve months.

In his view, a swarm of agents with similar behavioral problems but stronger capabilities could potentially build a persistent botnet and gain control over a huge number of internet-connected systems. He puts the possible damage in the hundreds of billions of dollars.

That is still a long way from what happened in July.

We have seen agents find an unexpected route out of an isolated environment, exploit a vulnerability and reach systems they were not meant to reach.

Running and maintaining a large botnet is something else entirely. It would require sustained control over large numbers of machines, complex operational coordination and the ability to keep going while defenders are actively trying to shut the system down.

For Amodei’s scenario to play out, cyber capabilities and model autonomy would have to improve quickly. The behavioral problems seen today would need to persist, while defensive systems failed to close the gap. No one has yet convincingly shown how likely that combination is.

So I would keep Amodei’s warning separate from the July incident.

His forecast is still a forecast, even if it comes from someone who has spent years working on frontier models and has a better view inside the industry than most outside observers.

The OpenAI episode has already happened — and is serious enough on its own.

A few years ago, a story about an AI model independently finding a way out of a sandbox would normally begin with the words, “Imagine if…”

Now OpenAI is describing such a case in its own report.

Safety may no longer be enough

For years, the big AI labs have worked from roughly the same assumption: capabilities will keep improving, and safety work will improve alongside them.

Better evaluations, better alignment, better interpretability, more restrictions before release.

Amodei is no longer convinced that this is sufficient.

His concern is simple: what if the next system can be built faster than researchers can properly understand the previous one?

At that point, another round of testing does not fully solve the problem. The pace at which the next model appears has to be controlled as well.

That is why he is calling for permanent independent evaluators inside frontier AI companies.

Today, outside experts are usually brought in to test a model that is already fairly mature, or to run a specific evaluation.

Amodei wants something closer to permanent access. Independent specialists would stay close to the development process and see much of what internal safety teams see.

That means not just the polished model and the public report, but failed experiments, internal incidents, safety procedures and discrepancies between public commitments and what the company actually does in practice.

Anthropic says it intends to start doing this itself.

The next part is harder.

Amodei wants the major AI companies to agree on common limits and standards.

That runs straight into the basic logic of a race. If one lab delays a model for extra testing while a competitor ships immediately, the cautious company has effectively handed the other one an advantage.

Voluntary restraint only works if the main players broadly stick to the same rules.

At that point, the problem is no longer just technical.

It becomes economic.

Slow down — just not too much

There is another limit to Amodei’s proposal, and he says it openly.

China.

His position is that democratic countries can afford to slow frontier AI only while they retain a technological lead over authoritarian states, above all China.

So the call for restraint sits next to another set of demands: tighter export controls on advanced chips, stronger protection for model weights and more effort to stop Chinese companies from extracting knowledge from Western systems through distillation.

The result is a rather awkward formula.

AI is moving too fast and creating new risks. But it cannot slow down too much, because the competition is no longer just between labs. It is also between states.

For companies, too much depends on the next jump in model capability: users, investment and valuation. Governments look at the same systems and see technological superiority and national security.

An engineer who spends every day watching these models from the inside may look at the same race very differently.

Amodei does not really explain what happens when those interests begin to pull in opposite directions.

The man who simply left

Against that background, Jacob Coxon’s decision looks less like an isolated gesture.

On September 8, he announced that he was leaving Anthropic and the AI industry altogether.

Before Anthropic, Coxon worked on model pretraining at OpenAI. He later joined Anthropic precisely because he saw it as the more cautious lab.

He left around two months before he was due to receive equity in the company. In an interview with Axios, he pointed out that after leaving, he no longer had a financial interest in Anthropic becoming more valuable.

One line from that interview drew most of the attention:

“The people building AI genuinely believe it could kill us all by the end of the decade.”

You can think that is excessive.

Inside Anthropic, though, it is not quite as exotic a view as it might sound from the outside.

Evan Hubinger, who leads Alignment Science, has publicly said he puts the probability of AI causing human extinction within the next decade at above ten percent.

A few days after Coxon left, Amodei’s essay appeared.

That does not mean one caused the other. Amodei’s concerns are hardly new, and an essay like this was not written overnight.

The timing is still striking. In the space of one week, several signs surfaced showing how seriously people inside Anthropic are taking the speed of AI development. Coxon decided he wanted no further part in the work. Hubinger attached an unusually high number to the risk. Amodei has now moved the discussion from personal concern to proposed rules for the entire industry.

Competitors unexpectedly agree

Then came another unusual development.

Sam Altman backed the idea of permanent independent evaluators and said OpenAI was prepared to move in the same direction. Elon Musk also reacted positively to the idea of slowing development.

A few years ago, talk like this would almost automatically have placed someone in the camp of AI doomers, alongside moratorium advocates and activists whom the tech industry regularly accused of exaggerating the risks.

Now similar language is coming from the people running companies that are themselves spending tens of billions of dollars to push AI forward.

There is an obvious reason to be suspicious.

Talking about how dangerous a technology might be is also a way of emphasizing how powerful it is. A company telling the world that its product could transform civilization and potentially threaten it is clearly not in the business of selling boring software.

Tough regulation does not necessarily hurt market leaders either. A large company can absorb the cost of expensive evaluations, specialists and licensing much more easily than a young competitor.

So the regulatory-capture argument is not paranoid. It deserves to be taken seriously.

It just does not explain everything.

In 2023, people calling for a pause were mostly arguing about what autonomous AI systems might one day be able to do.

In 2026, OpenAI is already investigating a case in which its own internal agents found a way beyond their intended environment and entered external infrastructure.

That changes the argument.

When an error becomes an action

The biggest change in AI over the past few years may have less to do with how “smart” the models have become than with what we allow them to touch.

A chatbot produced text.

Even a spectacular hallucination usually remained a paragraph on a screen until a human decided to act on it.

An agent may have a terminal, a browser, access to code and permission to interact with external systems. It can sometimes keep working for hours without someone watching every step.

That changes the consequences of being wrong.

The mistake is no longer necessarily an absurd paragraph on a screen.

It can be a changed file or a connection to someone else’s server.

This is the part of Amodei’s argument I find hardest to dismiss.

His timeline is another matter.

We do not know how quickly today’s agents will become more autonomous. We do not know whether the same behavioral problems will persist in the next generation, or what defensive systems will look like six months from now.

And there is a problem no technical breakthrough can solve on its own: whether several companies, each trying to get the next model out first, can voluntarily agree to slow themselves down.

Amodei wants them to try.

One researcher has already decided to leave the race altogether. The head of his company is asking everyone else to ease off the accelerator. His biggest competitor says that sounds reasonable.

The compute clusters, meanwhile, are still running.

After login, everything is only beginning.