The most significant detail in Amodei’s essay, We Must Pace The Frontier, was a reference to the Hugging Face incident, which occurred in July, when the rival tech company OpenAI revealed that an autonomous AI agent powered by its technology hacked a prominent database of AI models, Hugging Face. Coverage of the hack made it sound as though OpenAI’s model had gone “rogue”. But tech researchers have since said that OpenAI turned off the model’s safety mechanisms, gave it impossible tasks to complete and ran it 1,200 times. “That’s human decision-making,” the tech research Eryk Salvaggio wrote in an essay on Substack.
OpenAI then invited researchers from the non-profit institute METR to produce a report about the incident. As two legal experts observed in an interesting opinion piece for the Guardian, METR’s findings were constrained by its agreement with OpenAI, which prohibited investigators from accessing the underlying model that created the “rogue” agents.
“Imagine a burglar who is hallucinating and breaks into the Louvre,” says Aisha. “Then imagine you shared a tape with the police that played back what the hallucinating burglar was saying to himself while he broke into the Louvre, but the report didn’t actually include any information about what the burglar did. That’s basically what OpenAI did.”
In his essay, Amodei pointed to the Hugging Face incident as an example of the “catastrophic damage” that AI could cause in future. But industry critics are sceptical about what this incident can tell us, particularly since the only people who have examined it did so at OpenAI’s discretion. This reflects a broader problem where companies that claim to be developing products capable of catastrophic damage get to mark their own homework. “No other industry gets treated that way,” Aisha says. “If a nuclear company had a radioactive spill, you wouldn’t be letting it regulate itself.”
Is it all just hype?
Amodei’s essay has been treated as a doomsday warning. It was published just two days after Jacob Coxon, a 27-year-old researcher at Anthropic, quit his job and posted an alarming thread on X warning that “the people building AI earnestly believe that it could kill us all by the end of this decade”. Coxon is the latest in a long list of AI researchers who have been loudly quitting on X or making similar predictions. These self-styled whistleblowers don’t tend to leak documents that reveal useful information about the inner workings of AI companies, so their warnings of imminent apocalypse instead fuel speculation about the perils (and power) of AI.
This plays into tech companies’ hands. The AI narrative is dominated by what the tech scholar Lee Vinsel calls “criti-hype” – criticism that feeds, and is fed by, hype. The language used to describe AI can be deeply unhelpful. While China tends to compare AI to “electricity,” western AI companies compare it to the atom bomb, which makes Anthropic and its rivals seem less like companies making everyday decisions about how they choose to build particular tools, and more like Promethean stewards of a fourth Industrial Revolution. “The framing is, ‘we’re inventing fire’,” Aisha says. “And that means governments have to treat them like magical gods and let them set the rules on their mysterious creation.”
So why are AI firms calling for a slowdown?
Apparently, AI firms now believe the risks posed by their technologies are too great for business to continue as usual. But there are a number of ways that Anthropic (and its competitors) might gain from a slowdown. Anthropic is launching its initial public offering later this year, so it has an interest in presenting its products in as epochal and deadly a fashion as possible. The backlash against AI is growing, and the firm has cast itself as a more responsible AI creator, most recently by refusing the Pentagon’s requests to use its Claude model to power autonomous lethal weapons. Meanwhile, American AI firms face increasing competition from Chinese AI firms, and are desperate to retain their share of the market.
An industry-orchestrated slowdown could freeze the pace of AI development in China and might also help firms like Anthropic avoid the possibility of stringent government regulations by putting them on the front foot. One of the people who has called this out is David Sacks, co-chair of Trump’s council of advisers on science and technology, who said in response to Amodei’s essay: “demanding your preferred regulatory framework … will look like blackmail of the public and the political system”.
What should the UK government do?
To be clear, AI poses huge risks. Readers of First Edition will know that we’ve previously covered how the datacentres used to power AI demand huge volumes of water and energy. You can already use AI to generate sexualised “deepfakes” and child sexual abuse images, or make use of an open-source version of Palantir to monitor live camera feeds (meanwhile, Palantir has access to identifiable NHS England data). And experts have long been arguing that as AI becomes more advanced, it could pose existential risks.
All of these things are terrifying. But much of the AI narrative focuses on extinction-like events, the Terminator hypothesis, rather than dangers that are playing out right now. And many of the organisations that are supposed to make AI safer are closely aligned with the tech industry, such as Britain’s AI Security Institute, which receives funding from the industry, and doesn’t have the legal power to regulate AI or to compel developers to hand over their models. One of the institute’s co-founders, Matt Clifford, who lobbied for the government to loosen copyright laws, was recently forced to stand down from his role as chair of the UK’s Advanced Research and Invention Agency after taking a job with Anthropic … nice work if you can get it.
Recently, I’ve notice a creeping sense of fatalism in my conversations with friends about AI. Nobody (aside perhaps from government) consented to the volumes of energy, resource and information being fed into the giant AI machine. But as Aisha makes clear, the design of technology is a choice, not an inevitability, so there’s an argument for stubborn resistance to the idea there’s no alternative.
“We should be treating AI like any other industry, and regulating it like any other industry,” Aisha says. “Policy based on fear and vibes is just never going to work.”
Dario Amodei, co-founder and chief executive officer of Anthropic. Photograph: Bloomberg/Getty Images
