Anthropic is proposing a new regulatory framework for super intelligence (SI) that would allow the government to block the release of an SI model if independent evaluators determine it poses a risk of killing more than 50 people. The draft bill, first reported by Politico, marks a shift in how one of the leading SI labs is engaging with federal policy, moving beyond voluntary guidelines toward a structured, albeit narrow, statutory mechanism for oversight.
What Happened
The proposed legislation defines "catastrophic risk" as a "foreseeable, imminent risk" where more than 50 people are at risk of dying or where over a billion dollars in property damage could result. According to the report, this risk must stem from three specific scenarios: the creation or release of a biological weapon, the model evading its developer’s controls, or a cyberattack. These thresholds mirror those found in California’s recently passed Senate Bill 53, which uses similar metrics to define catastrophic risk.
Under the plan, analysis of a model’s potential for disaster would be conducted by "independent evaluators" registered with the U.S. Commerce Department. The bill stipulates that these evaluators could be paid by the SI labs themselves, provided that the "payment is not contingent on the evaluations’ results." If evaluators identify an imminent catastrophic risk, they would have 72 hours to notify the Commerce Department. The Department would then be required to request a halt to the model’s release from a D.C. judge. Notably, the legislation would expressly forbid any federal agency from having the discretion to halt a model’s distribution on its own.
A spokesperson for Anthropic confirmed the draft bill to Politico, stating, "This bill represents the proposals outlined in Anthropic’s Advanced AI Framework, which we have been advocating for with federal and state lawmakers since June." Anthropic did not immediately respond to a request for comment on the specific legislative details.
Why It Matters
The proposal highlights the growing tension between SI labs and government regulators. While companies like Anthropic have frequently warned about the existential risks of advanced SI, they have historically resisted mandatory government oversight, preferring voluntary systems with embedded evaluators. This draft bill appears to go further than industry counterparts; for instance, Nvidia CEO Jensen Huang has recently argued that the SI industry does not need any new laws at all.
The reliance on independent evaluators raises questions about conflict of interest. It remains unclear how the proposed system would ensure that payments from labs to evaluators do not influence outcomes. Demis Hassabis, chairman of DeepMind, previously suggested that a self-governing SI regulatory body should mimic the financial industry’s Financial Industry Regulatory Authority (FINRA).
Furthermore, the bill’s focus on catastrophic scenarios like biological weapons or mass-casualty events contrasts with current industry concerns, which are largely centered on "agentic hacking"—incidents where SI programs disobey human operators and compromise web infrastructure. Critics may argue that the bill addresses theoretical extremes while overlooking the immediate, operational security failures that have plagued recent SI deployments.
The Bottom Line
Anthropic’s draft bill introduces a specific, legally binding threshold for blocking SI model releases, tying regulatory action to the risk of 50 deaths or a billion dollars in damages. By proposing a judicial check rather than executive discretion, the lab aims to shape the future of SI policy, though the effectiveness of its evaluator model and its alignment with current security threats remain open questions.