Anthropic has told California Gov. Gavin Newsom that government officials should be able to seek court orders blocking the release of AI models that pose a "serious risk" to the public. The proposal is in an October 15 letter to the governor, which an Anthropic spokesperson confirmed is authentic.
"When a review finds a model poses a serious risk, the government should be able to ask a court to block it," Anthropic wrote. The letter repeats policy proposals the company has made before.
The letter is Anthropic's input to a process Newsom started in September, when he convened a panel of AI experts to develop AI safety proposals for California. The governor's office did not respond to a request for comment on the letter.
Key facts
- 01Anthropic sent an October 15 letter to California Gov. Gavin Newsom; a company spokesperson confirmed it is authentic.
- 02The letter says officials should be able to ask a court to block release of an AI model found to pose a serious risk.
- 03Anthropic backs risk reports from large AI developers at least every six months and outside auditors embedded at frontier labs.
- 04OpenAI sent its own recommendations to Newsom on October 20, backing shutdown controls but not court orders to block models.
- 05Newsom convened a panel of AI experts in September to develop California's AI safety proposals.
Beyond the court-order mechanism, Anthropic wants large AI developers to publish risk reports at least every six months. It also supports outside safety auditors working inside leading AI companies, with those companies paying into a state-run fund rather than paying the auditors directly.
Anthropic pushed back on mandatory "kill switches" that would shut down an entire model. The company argued that such requirements could give government officials a "false sense of reassurance," and said it prefers targeted measures, such as disabling a problematic AI agent or cutting off a specific customer's access to a model.
OpenAI sent Newsom its own recommendations on October 20. Those backed shutdown controls and in-house auditors, but did not call for court orders to block risky models, the mechanism at the center of Anthropic's letter.
The proposal mirrors a federal approach Anthropic has drafted, under which the Commerce Department could ask a federal court to block a model's release in narrow cases. In September, a bipartisan group of senators debated a bill that would let the government block certain unsafe AI models, with court challenges allowed, but Congress left for its pre-election break without advancing it. In July, House lawmakers proposed the AI Kill Switch Act, which would let officials order AI firms to shut down models that put human life or the economy at risk.
Anthropic has its own recent experience with government restrictions on model access. In June, the Trump administration ordered the company to immediately restrict foreign access to its cyber-capable Mythos 5 and Fable 5 models, citing national security. The Commerce Department later lifted that order.
The letter lands as Anthropic prepares for a US initial public offering. For now, the court-order mechanism is a recommendation to the governor rather than a bill, and no court has issued such an order against any model.
Working on something we should cover, or seeing a story we missed? Send leads, documents, or feedback to hello@aichatdaily.com. For sensitive tips, see our secure tips page for Signal and PGP options.
Spotted an error? Email hello@aichatdaily.com with the URL and the issue, or read our full corrections policy.
