Policymakers and commentators often treat large language models (LLMs) as if they were searchable repositories of personal data. The intuition is understandable: these systems train on massive corpora that may include personal information, and they occasionally generate outputs referencing real people.
But the analogy is still wrong. And policy built on it risks distorting both innovation and privacy enforcement.
I’ve written a new issue brief examining the empirical evidence on LLM memorization, distinguishing it from analytically separate phenomena such as hallucination and inference, and surveying how existing U.S. privacy law addresses these issues. The research points in a consistent direction: large language models do not store personal data like databases, memorization of personal information is atypical, and privacy risk arises primarily at the point of output and use, not from internal statistical representations.
Autocomplete, Not Archive
In machine-learning research, “memorization” has a precise meaning: a model reproduces training text verbatim or near-verbatim, usually under targeted or adversarial prompting.
Researchers do not agree on every boundary, but the throughline is clear. One paper defines “exact memorization” as token-for-token reproduction and “approximate memorization” as matching within a 10% edit distance; another similarly tests whether models can reproduce training-data text verbatim.
That usage is far narrower than in policy debates, where “memorization” often becomes shorthand for the broader worry that models “contain” personal data as discrete, retrievable objects.
They do not.
LLM parameters encode probability distributions over token sequences. As Michael Duan and his coauthors explain, the model is an autoregressive system that predicts the probability distribution of the next token, given a prompt. Nearly all outputs reflect probabilistic reconstruction, not retrieval of stored records. The distinction matters enormously for how the law should treat these systems.
Email Signatures, Not Secrets
Across multiple studies, verbatim memorization appears uncommon relative to the scale of modern training data. Even very large models trained on multi-trillion-token datasets show reproduction rates of roughly 1% to 4% in recent frontier-model evaluations, and newer model generations often copy less than earlier ones.
The strongest extraction methods rely on contrived prompts seeded with partial strings known to appear repeatedly in the training material. Without that scaffolding, random prompting produces little to no verbatim reproduction. A landmark extraction study by Nicholas Carlini and colleagues illustrates the point: out of more than 600,000 model generations, the researchers confirmed only 604 verbatim extractions, about a 0.1% rate.
Even those results required highly adversarial sampling procedures and independent access to the underlying training data to verify matches. A later large-scale analysis by Milad Nasr and coauthors reached similar conclusions, with only 0.03% to 1.4% of outputs containing recoverable training text.
Two empirical patterns stand out. First, duplication strongly predicts memorization. Strings that appear tens or hundreds of times in the training material—such as email signatures and boilerplate language—are far more likely to be reproduced. Removing repeated material from training datasets can reduce verbatim memorization by an order of magnitude.
Second, low-entropy strings—highly predictable text with few plausible continuations—are disproportionately memorized. Templated contact information and similar low-entropy content strings are easier to reproduce. High-entropy material, such as narrative prose or unique personal messages, rarely reappears in the same form.
These patterns indicate that memorization is localized. The individuals most at risk tend to have large digital footprints: public figures, academics, and others with SEO-dense online profiles whose information appears repeatedly across the web. Models rarely reproduce one-off personal information about private individuals unless it has been duplicated extensively.
Developers also deploy multiple safeguards to reduce memorization risk, including dataset curation, deduplication, red-teaming, classifier-based filtering, and decoding controls. Researchers have explored more aggressive interventions, such as “machine unlearning,” but those techniques remain technically immature and can degrade model quality. Mitigation therefore involves tradeoffs, yet existing tools can materially reduce observable memorization.
A Guess Is Not a Breach
Much of the confusion in policy debates comes from conflating memorization with two different phenomena: hallucination and statistical inference—outputs produced by pattern-based guesswork, rather than recalling a specific source. When a model says “John Doe is a lawyer in Chicago,” the statement may be a fabrication drawn from common name-plus-profession-plus-city patterns, not a retrieved record. Treating that kind of probabilistic guess as the unauthorized disclosure of factual personal data would impose deterrence wildly disproportionate to any harm, particularly when the asserted personal data was never part of the model’s training material.
A framework that treats hallucination as data leakage—as if the system were revealing protected information—would effectively treat statistical inference as the exposure of a stored record. The predictable result would be reduced social utility, as developers suppress useful generative features to avoid liability for outputs that do not reveal stored information. This problem differs from cases in which outputs combine publicly available facts about real individuals with false or distorted claims, which instead resemble accuracy-based or defamation-style harms.
Weights Are Not Records
Existing federal privacy statutes reflect a principle consistent with how LLM memorization actually works: liability attaches to disclosure, misuse, or failure to safeguard identifiable personal information within defined relationships and contexts. None of the major federal regimes treat the conversion of text into statistical model parameters as legally relevant storage.
The Health Insurance Portability and Accountability Act (HIPAA) regulates covered entities in the health-care system. The Fair Credit Reporting Act (FCRA) governs consumer-reporting agencies and eligibility determinations. The Children’s Online Privacy Protection Act (COPPA) addresses operators of child-directed services. The Gramm-Leach-Bliley Act (GLBA) covers financial institutions. The Video Privacy Protection Act (VPPA) targets video-service providers. Each statute identifies specific actors, specific data types, and specific prohibited conduct. Across all of them, the trigger is misuse or disclosure of identifiable data, not the existence of internal representations. Treating model weights as regulated “personal information” would stretch statutory definitions and likely invite challenges as inconsistent with statutory text and historically understood triggers, particularly in light of Loper Bright Enterprises v. Raimondo.
A general-purpose LLM developer therefore does not become a regulated entity under these statutes merely because a model learned from mixed written materials that happened to include health, consumer, or financial information. The legal risk arises at deployment: when a covered entity uses an LLM in a way that discloses identifiable information or incorporates protected data into eligibility decisions.
Even the Federal Trade Commission’s (FTC) more flexible “unfairness” authority requires substantial consumer injury that consumers cannot reasonably avoid. Internal model encodings do not meet that standard by themselves, because any injury depends on downstream output. The mere existence of model weights capable of producing text that is facially similar to personally identifiable information (PII) does not, standing alone, implicate the FTC’s privacy framework.
California’s California Privacy Rights Act (CPRA) is broader but points in a similar direction. The statute introduces data-minimization rules, sensitive-personal-information categories, and risk-assessment obligations. Yet California law still focuses on business practices, proportionality, and downstream sharing, rather than treating models as repositories of personal data. The CPRA does not expressly classify model weights as personal information, and its exclusion for publicly available information further narrows the set of regulated outputs.
Regulate the Harm, Not the Metaphor
The evidence supports a straightforward conclusion: rules that treat all generated personal data—whether real or fabricated—as if it were retrieved from stored records risk serious overdeterrence. They invite restrictions on generative systems even when the empirical risk of disclosing memorized personal information remains low and typically confined to duplicated, highly predictable material.
A more coherent approach would align legal obligations with observed behavior. Privacy law has long focused on disclosure, misuse, and failure to safeguard identifiable information in real contexts. That same logic fits here. Regulators should evaluate outputs and deployment—how information is used, shared, and protected in practice—rather than speculate about what internal statistical parameters might theoretically encode.
Conflating hallucination with disclosure, or probabilistic pattern-matching with record retrieval, misidentifies the harm. When models fabricate facts about real people, the concern resembles accuracy or defamation-style injury. When a covered entity uses a system to expose protected data, existing privacy law already applies. Treating models themselves as databases does neither.
Policymakers should resist regulating by analogy. Large language models are not databases; they are predictive systems trained on language patterns. Rules built on the database metaphor would do little to improve privacy protection, while discouraging socially valuable uses of generative tools. A framework centered on outputs, context, and safeguards can protect individuals—all without regulating mathematics as if it were a filing cabinet.
