Save StorySave this storySave StorySave this story
I’ve spent the past few days asking AI companies to convince me that the prospects for AI safety have not dimmed. Just a few years ago, it seemed that there was universal agreement among companies, legislators, and the general public that serious regulation and oversight of AI was not just necessary, but inevitable. People speculated about international bodies setting rules to insure that AI would be treated more seriously than other emerging technologies, and that could at least provide obstacles to its most dangerous implementations. Corporations vowed to prioritize safety over competition and profits. While doomers still spun dystopic scenarios, a global consensus was forming to limit AI risks while reaping its benefits.
Events over the last week have delivered a body blow to those hopes, starting with the bitter feud between the Pentagon and Anthropic. All parties agree that the existing contract between the two used to specify—at Anthropic's insistence—that the Department of Defense (which now tellingly refers to itself as the Department of War) won’t use Anthropic’s Claude AI models for autonomous weapons or mass surveillance of Americans. Now, the Pentagon wants to erase those red lines, and Anthropic’s refusal has not only resulted in the end of its contract, but also prompted Secretary of Defense Pete Hegseth to declare the company a supply-chain risk, a designation that prevents government agencies from doing business with Anthropic. Without getting into the weeds on contract provisions and the personal dynamics between Hegseth and Anthropic CEO Dario Amodei, the bottom line seems to be that the military is determined to resist any limitations on how it uses AI, at least within the bounds of legality—by its own definition.
The bigger question seems to be how we got to the point where releasing killer robot drones and bombs that identify and eliminate human targets wound up in the conversation as something that the US military would even consider. Did I miss the international debate about the merits of creating swarms of lethal autonomous drones scanning warzones, patrolling borders, or watching out for drug smugglers? Hegseth and his supporters complain about the absurdity of private companies limiting what the military can do. I think it’s crazier that it takes a lone company risking existential sanctions to stop a potentially uncontrollable technology. In any case, the lack of international agreements means that every advanced militia must use AI in all its forms, simply to keep up with its adversaries. Right now, an AI arms race seems unavoidable.
The risks extend far beyond the military. Overshadowed by the Pentagon drama was a disturbing announcement Anthropic posted on February 24. The company said it was making changes to its system for mitigating catastrophic risks from AI, called the Responsible Scaling Policy. It had been a key founding policy for Anthropic, in which the company promised to tie its AI model release schedule to its safety procedures. The policy stated that models should not be launched without guardrails that prevented worst-case uses. It acted as an internal incentive to make sure that safety wasn’t neglected in the rush to launch advanced technologies. Even more important, Anthropic hoped adopting the policy would inspire or shame other companies to do the same. It called this process the “race to the top.” The expectation was that by embodying such principles would help influence industry-wide regulations that set limits on the mayhem that AI could cause.
At first, this approach seemed promising. DeepMind and OpenAI adopted aspects of Anthropic's framework. More recently, as investment dollars ballooned, competition between the AI labs increased, and the prospect of federal regulation began looking more remote, Anthropic conceded that its Responsibly Scaling Policy had fallen short. The thresholds did not create the consensus about the risks of AI that it hoped it would. As the company noted in a blog post, “The policy environment has shifted toward prioritizing AI competitiveness and economic growth, while safety-oriented discussions have yet to gain meaningful traction at the federal level.”
Meanwhile, the competition between AI companies has gotten more cutthroat. Instead of a race to the top, the AI rivalry seems more like a bareknuckle version of King of the Mountain. When the Pentagon banished Anthropic, OpenAI rushed to fill the gap with its own Department of Defense contract. OpenAI CEO Sam Altman insisted that he entered his hasty deal with the Pentagon to relieve pressure on Anthropic, but Amodei was having none of it. “Sam is trying to undermine our position while appearing to support it,” Amodei said in an internal memo. “He is trying to make it more possible for the admin to punish us by undercutting our public support.” (Amodei later apologized for his tone in the message.)
All of this seems to point to a glum future where unfettered and dangerous AI proliferates. But the companies beg to differ. When I presented my bleak argument, they insisted that safety was as important as ever, despite the Pentagon’s affection for unreliable killer drones. ”I don't think the race to the top is dead,” says Anthropic’s chief science officer Jared Kaplan, urging me to shift my gaze from the battlefield and the marketplace to the research labs. “There are a lot of researchers at every lab that care a lot about doing the right thing. They want to see their research used for the betterment of humanity, and I think there is competition not just to make them more useful or capable, but also safer," he tells me.
OpenAI also tried to talk me off the ledge, pointing out that since it launched ChatGPT, a plethora of AI safety organizations have sprung up. And while the US hasn’t passed any federal regulations, the European Union is taking some steps to reign in AI. OpenAI, the company tells me, has more people working on safety than ever before. (It wouldn’t say, however, if the percentage of its now much larger workforce working on the issue has actually increased.) OpenAI’s chief strategy office Jason Kwon also argues that my observation that AI safety isn’t as prominent as it was a few years ago might be illusory. “The reason safety may seem less front and center is that other issues have popped up,” he says. “There's only so much you can hold in your head at any particular time. The safety question was a dominant question back in ‘23 and it's still an important question. But people are now also thinking about labor impact, how to use AI for economic growth, and how to distribute AI internationally so everyone has access.”
As for its Pentagon contract, OpenAI says that while it can’t control how the Department of Defense uses its models, the company has built in safeguards limiting its use for autonomous weaponry and other nefarious applications. But if Hegseth believed that removing those safeguards meant life or death for his soldiers, what’s to stop him from taking a step that it threatened Anthropic with—invoking the Defense Production Act to take over the company and remove the safeguards?
After talking about my fears with the companies, I was at least assured that safety still mattered to them. But I’m still skeptical that they will allow safety concerns to slow them down. Earlier this year, Anthropic's Amodei described this conundrum explicitly in an essay he published. “This is the trap,” he wrote. “AI is so powerful, such a glittering prize, that it is very difficult for human civilization to impose any restraints on it at all.” Happy Purim!
This is an edition of Steven Levy’s Backchannel newsletter. Read previous newsletters here.
