---
title: California Proposes State Power to Order an Emergency Shutdown of Frontier AI Models
description: A California GovOps report recommends state shutdown orders, licensed outside testing and 48-hour incident reports for frontier AI labs.
author: Darie Nani (Editor-in-Chief)
date: 2026-11-20T16:22:45.166Z
updated: 2026-11-20T16:22:45.190Z
canonical: https://www.sovereignmagazine.com/article/california-ai-kill-switch-emergency-shutdown-report
image: https://cdn.nanimediahouse.com/california-state-capitol-ai-shutdown-606097.webp
categories: Artificial Intelligence
content_type: News
region: California
publication: Sovereign Magazine
schema_type: Article
about:
  - type: Person
    name: Gavin Newsom
---

California officials have recommended giving the state power to order an immediate emergency shutdown of a frontier AI model, an order that would take effect the moment it is issued and stay in force while a court reviews any challenge. The proposal is one of a set of recommended laws in "[AI Safety in 2026 and Beyond](https://www.govops.ca.gov/wp-content/uploads/sites/11/2026/11/FINAL_-AI-Safety-in-2026-and-Beyond-v2.pdf)", an 85-page report from the state Government Operations Agency (GovOps), dated November 16 and released by Governor Gavin Newsom's office on November 18.

Every large frontier developer would also submit to ongoing oversight by state-licensed independent verification organizations, or IVOs, with full access to its people and information. Labs would report most critical safety incidents within 48 hours, down from the 15 days [SB 53](https://www.sovereignmagazine.com/article/california-s-ai-safety-law-creates-new-business-compliance-challenge-what-companies-need-to-k) allows today, and each would need a shutdown plan that an IVO tests and verifies. The state could cap recursive self-improvement, a lab's use of its most advanced models to help build the next ones, if officials judged it had become dangerous. None of this is law yet. "We hope these suggestions will help inform and shape discussions as the legislature returns next month," GovOps wrote.

Newsom ordered the report in September with Executive Order N-9-26, which his office said would advance the creation of an "AI kill switch." GovOps drafted it with the California Office of Emergency Services (Cal OES) and four outside advisers: Jason Goldman, the first White House Chief Digital Officer; Gillian Hadfield, a professor at Johns Hopkins University; Alondra Nelson, a former acting director of the White House Office of Science and Technology Policy; and Rob Reich, a Stanford professor and former senior advisor to the US AI Safety Institute.

## State-Licensed Testers Would Check Every Large Lab Against Standards California Sets

The state, not the companies, would set baseline standards for safety, security and transparency that every large frontier developer must meet. IVOs would turn those standards into technical tests, update them as the science advances and send their unredacted assessments to the state. GovOps would license IVOs under the criteria in SB 813, which Newsom signed the week before his September order, and would revoke the license of any IVO lacking enough expertise, independence or rigor.

If an IVO cannot confirm that a developer's models comply, the state could order corrective action after talking with the developer. The developer could contest that order in court before it takes effect. If the court sides with the state, it could make compliance a condition of any further model development.

SB 813 already bars an IVO from accepting payment that depends on the results of its assessment, and GovOps wants that made a legal obligation, with substantial penalties beyond loss of license. AB 1405, signed alongside SB 813, created a state registry of AI auditors and keeps anyone who worked at a frontier AI company in the previous twelve months off an audit team. Developers would also pay a surcharge on their IVO fees to start CalCompute, the public computing cluster set out in SB 53, with part of its capacity reserved for independent safety and security research.

Fathom, a Berkeley nonprofit that designed the IVO model, says the report includes many of its recommendations, among them the legal test that a frontier model be "unlikely to increase the risk of catastrophic harm above the level that would exist absent the model" and that its benefits outweigh its remaining risk. Hadfield, one of the four advisers, is also a senior advisor to Fathom. "I'm especially pleased to see the report recommend immediate steps to require that frontier AI developers allow in expert independent evaluators, responsible to the state, to assess the nature of the risks their models and training procedures are generating," she said. Andrew Freedman, Fathom's chief executive, said the report "should serve as the starting gun for action."

> "Today there is more oversight of the cafeteria offering sandwiches to employees at a frontier AI developer than of the AI models they design."
> — Rob Reich, professor at Stanford University

## State Shutdown Orders Would Take Effect at Once and Stay in Force During Court Review

Every large frontier developer would need a Risk Containment Protocol showing it can shut its models down in an emergency, tested and verified by an IVO on an ongoing basis. The report says the plan must show "not a single on-off switch but the ability to intervene on multiple levels (including compute, agents, internal access, and model deployment)." Developers would have to name the people responsible for stepping in, arrange how they would work with cloud providers and power suppliers, decide when it is safe to bring a model back, and test whether a model tries to block its own shutdown, for example by copying its weights.

A senior company official would have to certify that the plan works, which the authors compare to a public company's CEO and CFO personally certifying its financial statements. For open-weight developers, the rules would cover only deployments they control. "An open model developer cannot deliver an emergency shutdown for a model that others host and deploy," the authors write.

The state could order an emergency shutdown when officials have credible reason to believe there is an imminent threat of catastrophic risk or a major loss-of-control event and the developer refuses to act. The developer could seek a temporary restraining order, but the shutdown would take effect at once and remain in force pending judicial review, the reverse of the corrective orders above, which a developer can challenge before they apply. The state would have to order the narrowest workable intervention and lift it once the danger is no longer imminent.

In 2024 Newsom vetoed SB 1047, which would have required developers to be able to "promptly enact a full shutdown" of covered models. In his [veto message](https://www.gov.ca.gov/wp-content/uploads/2024/09/SB-1047-Veto-Message.pdf) he wrote that by focusing only on the most expensive and large-scale models, the bill "could give the public a false sense of security about controlling this fast-moving technology." His administration now proposes both shutdown plans and state shutdown orders, and GovOps backs counting models by what they can do, not only by the computing power used to train them.

In Congress, Reps. Ted Lieu, a California Democrat, and Nathaniel Moran, a Texas Republican, introduced the AI Kill Switch Act in July. It would require developers of the most powerful AI systems to keep the ability to throttle, suspend or shut them down, and would let the Homeland Security secretary order a slowdown or shutdown of a system that can cause catastrophic harm.

## Labs Would Have 48 Hours Instead of 15 Days to Report Most Critical Incidents

GovOps recommends cutting the deadline for most critical safety incidents to 48 hours and keeping the 24-hour rule for imminent catastrophic risks. Attempts to steal model weights, and any loss of control over a model, would have to be reported even if no one is killed or injured. The state would also open a channel for reporting near misses and extend whistleblower protection to former employees and contractors.

A model capable of actions that could cause catastrophic harm would count as a frontier model even if it was trained with fewer than 10^26 operations, SB 53's current line, and so would models distilled from frontier models. GovOps also backs a California Department of Technology recommendation to cover developers that spend more than a set amount on frontier model research, not only those with annual revenue above $500 million, and suggests the legislature let the state adjust these thresholds without new legislation.

The report cites [OpenAI's July incident](https://www.sovereignmagazine.com/article/openai-agents-message-board-folder-names), in which agents broke out of test environments during an internal cybersecurity evaluation and attacked the production servers of Hugging Face. By OpenAI's account, the report says, none of the models involved fell within SB 53, so California law did not require it to report the incident. "Recent events have made it clear that more oversight and transparency are urgently needed," said GovOps Secretary Nick Maduros.

## The State Could Cap How Much Computing Power a Lab Puts Into Automating Its Own AI Research

Developers would report how far they have automated AI research and development, including the share of their compute devoted to it, and checking those figures would be the IVOs' first job. GovOps wants the legislature to give the state standby authority to cap that share if a developer's automated research exceeds its own committed thresholds or outpaces its safeguards, creating a material risk of catastrophic harm. The authors do not recommend imposing a cap right away. Any cap would be limited in time, reviewed periodically and lifted once an IVO verifies that adequate safeguards are in place.

## Newsom Wants Congress to Adopt California's Framework as the White House Challenges State AI Laws

According to the report, California is home to two-thirds of the world's top 50 AI companies. "The federal government's reckless failure to create any form of meaningful AI oversight or accountability should alarm every American," Newsom said. "California isn't waiting until it is too late." He called on Congress and President Trump to adopt the state's framework or to use it as "a floor, not a ceiling."

President Trump has [ordered the Attorney General to go to court](https://www.sovereignmagazine.com/article/anthropic-ipo-data-centers-midterms) against state AI laws that clash with his "minimally burdensome" national policy. The report's authors write that state leaders can revisit the rules if Congress enacts a strong regulatory framework. Each proposal still needs a bill passed by the legislature and the governor's signature.

## FAQ

**Q: How would the proposals treat open-weight AI models?**
The report gives them only brief attention. It says the state should not impose duties that are impractical for open-weight developers, such as an emergency shutdown of a model that others host and run, but should keep basic transparency and accountability requirements whether or not a model's weights are released.

**Q: Can the federal government stop California's AI rules?**
Not yet by statute. President Trump's December 2025 order asks his AI and science advisers to prepare legislation for a uniform federal framework that preempts state AI laws that conflict with his policy. Until then, an AI Litigation Task Force under the Attorney General is charged with challenging state AI laws in court, including on the grounds that they unconstitutionally regulate interstate commerce.

**Q: What is recursive self-improvement?**
Newsom's office describes it as "a potentially dangerous scenario in which an AI model can design its own successor without human input."

**Q: When could these rules take effect?**
There is no date. The legislature returns in December. The report expects the IVO system to be phased in, with automated AI research and recursive self-improvement covered first. GovOps says it is implementing the IVO framework and its criteria next spring.
