Explainer / Published 2026-10-30

Anthropic asks Newsom to let California courts block high-risk AI models

In an October 15 letter to Governor Gavin Newsom, Anthropic proposed court injunctions, state-funded outside auditors and six-monthly risk reports. OpenAI backs auditors but not court orders.

A dark committee room with a sealed plain envelope on a bare wooden table, a low railing in soft focus behind it and cold window light from the left.
Conceptual illustration of a policy letter awaiting a decision; it does not depict Anthropic, Governor Newsom, a court or a real document.

THE SHORT VERSION

  • Anthropic wrote to California Governor Gavin Newsom in a letter dated October 15 arguing that when a review finds a model poses a serious risk, the government should be able to ask a court to block it.
  • The proposal routes independent evaluators' findings to California's attorney general, who could seek a court order pausing a model's release, its internal use or further work on it.
  • Anthropic also wants outside auditors paid through a state fund and risk reports at least every six months, and it argued that a law focused only on kill switches could give false comfort.

A request for a court, not a regulator's order

The letter is dated October 15 and responds to an executive order Newsom signed in September, which set up a working group to recommend how to strengthen California's AI safety law.

Anthropic's answer is that a shutdown power is useful but insufficient, and that the stronger mechanism is a judicial one.

From an evaluator's finding to an injunction

The mechanism Anthropic describes has three steps. Independent evaluators would report risks to California's attorney general. If the attorney general saw a near-term risk of catastrophic harm, they could ask a court to pause the model. A court order could then stop the model's release, its use inside the company, or further work on it.

That structure keeps a regulator from being the final decision-maker. It also puts a burden of proof in front of a judge rather than in a rulebook: the state would have to persuade a court that the evaluated risk is serious and near-term, and the company would have a forum to contest it. The letter said that when a review finds a model poses a serious risk, the government should be able to ask a court to block it, according to The Next Web.

Auditors inside companies, paid from a state fund

The second part of the proposal concerns who does the checking. Anthropic wants outside safety auditors to work inside leading AI companies, with firms paying into a state-run fund rather than paying the auditors directly. That arrangement is meant to remove the obvious conflict that appears when the company being examined also signs the examiner's invoice.

Anthropic also proposes that large developers publish risk reports at least every six months. Neither idea comes with an enforcement mechanism in the letter as described, and the reporting interval is a floor rather than a schedule tied to releases.

Why Anthropic rejected a kill-switch-only law

On kill switches, Anthropic agreed that every leading AI company should be able to shut down its models quickly. It then argued that a law focused only on shutdowns could give false comfort, because the dramatic remedy may not match the actual failure modes operators see day to day.

Instead it backed narrower steps: turning off a single AI agent, or cutting off one customer's access. Those are the interventions that a company running models in production can apply without stopping an entire service, and Anthropic's argument is that they are more likely to be used than an emergency stop.

OpenAI backs auditors, not court orders

Anthropic is not the only company advising the governor. OpenAI sent its own recommendations to Newsom on October 20, backing shutdown controls and in-house auditors but not court orders to block risky models, according to Politico's reporting as summarised by The Next Web. The split is narrow but real: both companies accept external scrutiny, and they disagree on whether a judge should be able to stop a release.

Anthropic is making a similar case in Washington. A draft federal bill it has circulated would let the Commerce Department ask a federal court to block a model's release in narrow cases, and House lawmakers have proposed an AI Kill Switch Act that would let officials order companies to shut down dangerous models. California's working group is the state-level version of the same argument.

What the letter settles and what it does not

Nothing has been decided. The letter is advice to a working group set up by the September order, and no bill, rule or court ruling has followed it. Newsom's office had not responded when the reporting was published, so the governor's position is unknown.

The letter also does not define what counts as a serious risk or a near-term catastrophic harm, which are the terms the whole mechanism turns on. Those definitions would have to be written into law before a court could apply them. Related coverage on this site includes the House bill setting a 48-hour clock for AI incident reports, Anthropic's three-tier cyber verification programme and the DOJ's questions about Nvidia's Hugging Face deal. Each is a different lever — disclosure, access control, competition review — and none of them is a court injunction over a model release.

A company asking for more scrutiny of itself

Anthropic's case is unusual because it asks for more outside scrutiny of the systems the company itself builds. The letter proposes a court that could block a model and auditors paid from a state fund, and the company says both are needed because a shutdown power alone is too blunt. A lab arguing for that kind of oversight is making a narrower claim than the headline suggests. The same letter asks for risk reports at least every six months and for outside auditors paid from a state-run fund.

That history explains the shape of the proposal. Anthropic is not asking for a regulator to be able to switch a model off at will; it is asking for a court, a standard of near-term catastrophic harm and a defined set of remedies. Whether that is stronger or weaker than a kill-switch law depends on how the terms are written, and the letter, as reported, does not write them.

Sources & reporting limits

  1. Anthropic tells Governor Gavin Newsom courts should be able to block risky AI models
  2. Anthropic 致信加州州长纽森:法院应有权阻止严重风险 AI 模型
Keep the thread.

Follow Policy & safety to find related stories on your next visit.

THE WEEKLY EDIT

The week, made clear.

Stories worth understanding, brought together in one weekly edition.

Share this page

Instagram, TikTok & other apps

Copy the page link, then paste it where your chosen app supports links. You choose whether to send or publish.