Anthropic has put forward a legislative proposal aimed at governing how the U.S. government responds to AI models capable of mass casualties, and the fine print has raised eyebrows among policy watchers. The bill's headline provision would let regulators intervene if a model poses a risk of killing 50 or more people. But buried in the structure of the proposal is a requirement that federal agencies obtain judicial approval before they can actually block that model from shipping, effectively adding a courtroom hurdle to what would otherwise be a direct regulatory power.
The arrangement marks an unusual stance for a company that has built much of its public identity around being the safety-conscious alternative in the AI industry. Anthropic has long argued that frontier AI systems carry genuine catastrophic risk and has positioned itself as an advocate for guardrails that other labs resist. A proposal that constrains the government's own ability to act quickly against a dangerous model, even while invoking a dramatic casualty threshold, complicates that narrative and invites questions about whether the bill is designed more to shape the terms of regulation than to strengthen it.
The timing is notable. Washington has spent much of the past two years debating how aggressively to regulate frontier AI, with industry players frequently pushing back against rules they see as slowing deployment or favoring rivals with more regulatory resources. By bringing its own detailed bill to the table rather than simply lobbying against others' proposals, Anthropic is attempting to set the baseline for what AI safety legislation looks like, rather than react to rules written by lawmakers or competitors.
That strategy carries real stakes. If a company-authored framework becomes the template Congress works from, the specifics of enforcement mechanisms, like requiring judicial review before a shutdown order takes effect, could end up locked into law. Critics of such provisions typically warn that court processes can be slow, and a model capable of catastrophic harm might not wait for litigation to run its course. Supporters would counter that due process protections prevent agencies from making unilateral, potentially overbroad decisions that could stifle legitimate AI development based on speculative risk.
The proposal lands at a moment when public trust in AI companies' self-assessment of risk is already under strain, with dueling narratives from across the industry about how dangerous current systems really are. How lawmakers respond to Anthropic's framing, and whether they add tighter enforcement powers or keep the judicial check intact, will say a lot about who ultimately gets to decide when an AI model is too risky to release.