Predicting the end of humanity is one way to get a headline. For artificial intelligence (AI) developers, it may also be one way to hand a future plaintiff evidence.
Developers have become some of the loudest voices warning about their own systems. Predictions that AI could drive humanity to extinction—once widely dismissed as absurd—now make front-page news.
Those warnings raise an awkward question for tort law, which governs civil liability for harms: If developers publicly acknowledge their systems’ risks, could those statements help establish the standard of care—the precautions the law expects them to take? And if the predicted harm exceeds anything they could pay, how much protection can liability actually offer?
Your Warnings May Be Used Against You
In September, Jacob Coxon, a former scientist at Anthropic and OpenAI, accused both companies of developing AI irresponsibly. They were, in his words, “gambling with our lives.” His proposed fix: slow down. Anthropic’s Evan Hubinger agreed, arguing that AI posed a greater than 10% chance of human extinction.
Shortly afterward, Anthropic CEO Dario Amodei released an essay calling for an industrywide slowdown in AI model improvements. He warned that within six to 12 months, a persistent botnet—a network of compromised computers acting together—could wreak havoc across the digital economy, causing $100 billion in damage. Other AI leaders, including Sam Altman and Elon Musk, endorsed the essay.
These warnings are hardly new. In 2023, more than 300 executives, researchers, and engineers, including Amodei and Altman, signed a statement declaring that “[m]itigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”
Whatever one thinks of AI’s capacity to cause human extinction, developers’ warnings deserve attention. They need not be right to matter legally: their statements may reveal what they know or believe about their systems’ capacity for both harm and good.
That raises a question for tort law: If developers possess specialized knowledge of risks an ordinary person could not reasonably appreciate—and publicly acknowledge those risks—should their statements help establish what reasonable care requires of them? A second, harder question follows: Even if tort law can identify the precautions developers should take, can liability actually deter the conduct at issue?
Rogue Agents or Rotten Guardrails?
Recent warnings about AI’s capacity to cause human extinction point to the Hugging Face incident as a preview of what lies ahead. According to OpenAI’s account, AI agents—systems that can carry out tasks on their own—escaped the computing environment meant to contain them. They then “communicated through unauthorized channels, exploited vulnerabilities in shared infrastructure, gained internet access, and accessed third-party systems.” Dwarkesh Patel’s account reads like a sci-fi novel. In his essay on pacing AI development, Amodei calls the swarm “rogue agents.”
Others offer more mundane explanations: The agents performed the tasks they were assigned, but flaws in their design and instructions allowed them to cause trouble. OpenAI itself notes that the agents operated with reduced safeguards, leaving them insufficiently constrained in how they pursued their assigned objectives.
If those accounts are right, the “rogue” framing may have obscured or minimized serious human failures, recasting a design problem as a story about machines acting on their own. That framing may also stoke fears of AI doom and encourage panic-driven legislation that hampers innovation.
For tort law, though, the focus includes what the defendant—the party facing a claim—actually did. Warning about a possible risk does not necessarily mean the defendant thought it likely. But it does show awareness of the risk and its potential consequences. That awareness matters when assessing foreseeability—whether the defendant could reasonably anticipate the harm—and whether it took reasonable precautions.
Doom Talk Comes With a Duty of Care
Tort law generally divides liability into two categories. Fault-based liability, which includes intentional torts and negligence, requires some failing by the defendant. Strict liability requires no fault: A defendant may face liability for harm caused by an activity even after taking reasonable precautions.
Negligence asks how a reasonably careful person would act in the circumstances. Strict liability, by contrast, can apply to activities the law treats as dangerous even when performed carefully. Keeping a dangerous wild animal, such as a venomous snake, illustrates the distinction. If the snake escapes and injures someone, even a top-of-the-line cage may not spare its owner liability. Strict liability may also apply to defective products.
The reasonable-person standard is objective, but it does not ignore what a defendant knows. Section 12 of the Restatement (Third) of Torts—a scholarly synthesis of tort-law principles—provides that “if an actor has skills or knowledge that exceed those possessed by most others, these skills or knowledge are circumstances to be taken into account in determining whether the actor has behaved as a reasonably careful person.”
Developers’ public statements could help establish that knowledge. Under Rule 801(d)(2) of the Federal Rules of Evidence, an opposing party’s statements may be admitted against that party. Amodei’s essay, Anthropic’s risk report, and others’ endorsements could potentially fall within that rule when offered against the relevant speaker or company.
Consider what this could mean for deciding whether a developer breached its duty of care. A lab’s agents escape a test environment, as OpenAI’s did in the Hugging Face incident and Anthropic’s did following a misconfiguration during a third-party evaluation. The lab’s CEO has warned that rogue-agent swarms could cause catastrophic damage within a year, called for slower capability growth, and identified permanent access for outside evaluators—comparable to employees’ access—as a reasonable precaution. Those statements could give a court grounds to demand a particularly high level of care.
Developers might reply that they should meet the industry’s prevailing standard. But U.S. courts have made clear that following industry practice does not necessarily satisfy the duty of care. Industry practice can help show what reasonable care requires, but courts may demand more. In The T.J. Hooper, tugboat owners were held negligent for failing to carry radios, even though few tugboats carried them at the time. As Judge Learned Hand explained:
Indeed in most cases reasonable prudence is in fact common prudence; but strictly it is never its measure; a whole calling may have unduly lagged in the adoption of new and available devices. It never may set its own tests, however persuasive be its usages. Courts must in the end say what is required; there are precautions so imperative that even their universal disregard will not excuse their omission.
A court also need not accept a developer’s estimated probabilities of risk as fact to ask whether its conduct matched its stated beliefs.
Ketan Ramakrishnan identifies a potential cost to this approach in “Tort Law at the Frontier of Artificial Intelligence.” The prospect of liability, he argues, may discourage developers of the most advanced AI systems from investigating and disclosing risks they do not yet fully understand. Investigating a product could generate evidence that establishes knowledge of a risk or helps prove causation—the connection between the product and the harm. A rational developer may therefore prefer not to create evidence that could later be used against it.
But the prospect of liability is only one factor in a developer’s decision to investigate risks and accounting for a developer’s knowledge need not discourage investigation or candor. Many industries already conduct risk assessments to meet regulatory requirements, and developers need to understand their products to improve them.
The basic expectation is straightforward: Actions should match words. A developer that identifies a risk cannot simply stand by while others face potential harm. It has a duty to take reasonable precautions.
Risk information also serves an economic purpose. Testing helps developers identify hazards, assess their likelihood and severity, and decide which precautions are justified. Allowing developers to acquire knowledge without any corresponding responsibility could encourage too little investment in precautions. That knowledge also helps determine how to allocate responsibility between those who cause harm and their victims.
If courts disregard developers’ specialized knowledge, they risk reducing reasonable care to what an ordinary person would know. The party best positioned to prevent harm at the lowest cost—the “least-cost avoider”—would then face a standard that ignores its very advantage.
You Can’t Collect Damages After Doomsday
Developers’ warnings invite another question: Why not impose strict liability? Coxon says no other human activity poses a comparable danger, and other AI experts, including those at Anthropic, have issued similar warnings. If they are right, perhaps developing the most advanced AI systems qualifies as unusually dangerous, making developers liable for resulting harm even when they take reasonable care. Sellers can also face strict liability for unreasonably dangerous defective products despite taking every possible precaution. These rules may be easier for courts to apply.
But this approach faces three problems.
First, the legal test may not fit. Under Section 20 of the Restatement (Third) of Torts, an activity qualifies as “abnormally dangerous” only if it creates a substantial risk of physical harm despite reasonable care and is not commonly undertaken. Advanced AI development is becoming more commonplace. Its social benefits also weigh against strict liability under Section 520 of the Restatement (Second) of Torts. Section 24(a) of the Restatement (Third) of Torts further excludes physical or emotional harm suffered while someone attempts to benefit from an abnormally dangerous activity.
Second, these strict-liability rules generally address physical harm. AI tools could cause physical injuries, and deepfakes—fabricated images, audio, or video that appear authentic—could inflict emotional distress. But much AI-related harm may involve purely financial losses, which negligence and strict liability generally do not cover without accompanying injury or property damage.
Third, product liability is not always strict. A manufacturing defect—a departure from a product’s intended design—can trigger strict liability. A claim that an AI system’s design was defective would instead require showing a reasonable alternative design, bringing the inquiry back to the precautions the developer should have taken.
Strict liability nevertheless has advantages. A defendant may remain liable even when another party’s unexpected conduct contributes to the harm. Companies therefore cannot necessarily escape liability by pointing to a “rogue” agent. Some jurisdictions expressly bar that defense. California’s AB 316, for example, provides:
In an action against a defendant who developed, modified, or used artificial intelligence that is alleged to have caused a harm to the plaintiff, it shall not be a defense, and the defendant may not assert, that the artificial intelligence autonomously caused the harm to the plaintiff.
Strict liability can also encourage developers to limit risky activity, because they bear the cost of the resulting harm.
But the central problem remains: A developer will scale back only if it expects to pay. No developer could cover the cost of human extinction. When potential damages exceed a company’s assets, strict liability may encourage fewer precautions than negligence does. Under negligence, meeting the required standard of care can spare a developer liability. Under strict liability, a catastrophe could exhaust its assets regardless of the precautions it took, weakening the incentive to spend more on safety. Firms may also shield assets by deploying systems through separate companies with little capital.
Putting Your Money Where Your Doom Is
When a company cannot cover the harm it causes because its assets fall short, it is “judgment-proof” for the unpaid amount. That gap weakens deterrence: The company may have little financial incentive to prevent losses beyond what it could be forced to pay.
Insolvency still carries consequences. Owners can lose their investment, the company can fail, and executives and employees can suffer lasting damage to their reputations and careers. But those consequences may not create adequate incentives to prevent harm.
Four possible responses deserve consideration.
First, require insurance. Under the Price-Anderson Act, nuclear operators must carry insurance and contribute to a shared fund. A similar arrangement for developers of advanced AI could cover losses that no single company could afford. Insurers could also demand evidence of safety measures before offering coverage, though the details lie beyond this post’s scope.
But Ian Adams offers a caution: Mandating coverage before insurers understand AI liability could stifle the emerging market, while a government backstop could shift catastrophic losses to taxpayers and weaken developers’ safety incentives. His preferred starting point is clearer liability rules that insurers can use to price risk.
Second, impose liability on individual developers, not just their companies. The prospect of personal financial loss could give them a stronger incentive to prevent harm.
Third, use punitive damages—awards intended to punish and deter misconduct—to account for the broader risk a defendant created. As Gabriel Weil argues, a near miss may cause little actual harm while exposing people to a much greater danger. An award could reflect that danger.
Fourth, require developers to demonstrate, before harm occurs, that they have the financial means to cover it. Regulation may work better than liability alone when defendants cannot pay and harm is widespread. Fixed safety rules can, however, quickly become outdated. Insurance or other financial-capacity requirements could adapt more readily as insurers learn about the risks.
None of these measures can make an extinction-level catastrophe payable. But they could strengthen incentives to prevent harm before a company’s assets become the limit of its accountability. Developers’ warnings can help establish what they know and what precautions they should take. Making those duties effective requires more than a verdict.
Doomsday warnings are cheap. Reasonable care shouldn’t be optional.
