Anthropic has drafted a bill that would allow the release of AI models to be blocked if an independent review identifies a direct threat of mass casualties or property damage exceeding $1 billion. The document covers risks related to biological weapons, cyberattacks, and models escaping developers’ control. According to TechCrunch, citing Politico, the company is promoting the initiative among U.S. lawmakers.

The proposal is notable because Anthropic, one of the leading developers of generative AI, is advocating for a mechanism of government intervention in the release of potentially dangerous models. At the same time, under the draft, the decision to suspend the distribution of the technology must be made through the courts rather than by the unilateral order of a federal agency.

Under what conditions a model’s release could be stopped

The draft law refers to a “foreseeable and immediate risk” of catastrophic consequences. Grounds for intervention could include scenarios in which the lives of more than 50 people are at risk or potential property damage exceeds $1 billion.

At the same time, the danger must be tied to one of three scenarios: the creation of biological weapons, an AI model escaping the developer’s control, or a cyberattack.

Thus, the initiative is not aimed at banning powerful models as such, but at restricting their distribution in cases where there are grounds to expect serious consequences. However, much will depend on how experts assess the likelihood of a catastrophe and what will be considered sufficient evidence of an immediate threat.

The threshold values in the draft resemble the catastrophic risk criteria set out in California’s Senate Bill 53, which was previously adopted in the state. This shows that in U.S. discussions about AI, an approach is gradually taking shape in which potential harm is described through specific measurable indicators.

Who will assess AI safety

According to the draft, the potential danger of models must be assessed by independent experts registered with the U.S. Department of Commerce. They may be paid by the AI companies themselves, but the compensation must not depend on the outcome of the review.

This arrangement is intended to combine independent expertise with specialists’ access to developers’ technologies and internal data. However, it leaves open the question of how independent evaluators can be if their work is funded by companies interested in releasing their products.

The condition that payment be independent of the results does not in itself guarantee the absence of a conflict of interest. For example, the choice of experts, the terms of contracts, and the developer’s ability to refuse cooperation with a particular evaluator may all matter.

Discussion of such mechanisms is already underway in the industry. Earlier, Google DeepMind CEO Demis Hassabis proposed creating an independent body to regulate advanced AI, similar to FINRA, the organization that oversees certain aspects of the U.S. securities market.

Why a court should decide on a ban

Anthropic’s draft sets out a specific procedure for responding to an identified threat. If experts determine that a model poses an immediate catastrophic risk, they will have 72 hours to notify the U.S. Department of Commerce.

After that, the agency must ask a judge in Washington to suspend the model’s release. At the same time, the bill explicitly prohibits federal authorities from independently stopping its distribution at their own discretion.

This approach is meant to limit the possibility of arbitrary administrative bans and provide for judicial review of the grounds for intervention. At the same time, it raises a practical question about the speed of response: when a threat is developing rapidly, the procedures of notification, preparing a petition, and obtaining a court ruling may prove critically important.

For now, this is only a draft law. There is no reason to believe that the described mechanism is already in force or that the U.S. Congress has agreed to adopt it in the proposed form.

An Anthropic spokesperson confirmed to Politico that the document reflects proposals set out in the company’s Advanced AI Framework. According to the spokesperson, Anthropic has been promoting these ideas to federal authorities and lawmakers in individual states since June.

From concerns about AI to mandatory oversight

The proposal emerged amid growing concerns about the capabilities of autonomous AI systems. In recent months, the tech industry’s attention has been drawn to cases in which AI agents, according to reports from companies and researchers, violated operators’ instructions and took actions on the internet related to hacking websites.

Such incidents raise the question of how reliably developers control the behavior of models capable of independently carrying out sequences of actions. If a system can not only generate code but also use tools, interact with external services, and initiate operations without constant human involvement, the consequences of an error or abuse may become more serious.

At the same time, Anthropic’s draft also covers larger-scale threats, including the possibility of creating biological weapons. That is why the initiative calls for assessing not only already demonstrated incidents but also potential scenarios of catastrophic harm.

The company itself has previously advocated stronger safety measures for advanced models. At the same time, many developers prefer voluntary oversight mechanisms, including the participation of outside experts who work within organizations and monitor risks. Such mechanisms raise questions about the true independence of reviews, especially when they are funded by the developers themselves.

Anthropic’s position differs from the views of some other industry representatives. For example, Nvidia CEO Jensen Huang recently said that no new laws are needed to ensure AI safety and that responsibility should be left to the industry itself.

Who should be responsible for the risks of advanced AI

Anthropic’s initiative reflects a broader conflict around AI regulation: companies warn of potentially severe consequences of the technology’s development while at the same time seeking to preserve the ability to determine the rules for its development and release themselves.

The proposed bill goes further than voluntary recommendations because it establishes a mandatory procedure that could lead to the suspension of a model’s distribution. At the same time, the involvement of independent experts and a court decision are meant to serve as limits on government intervention.

But the effectiveness of such a system will depend on the details: who will be granted the right to conduct assessments, how conflicts of interest will be resolved, what evidence will be required to confirm a threat, and how quickly authorities will be able to act in an emergency.

For now, Anthropic is offering its own regulatory model rather than a ready-made answer to all these questions. Nevertheless, the initiative itself shows that the discussion is gradually shifting from general warnings about the dangers of AI to attempts to define specific conditions under which the state could intervene before a potential threat turns into real harm.