Here is a scenario the law has no answer for yet. The government lawfully collects an enormous volume of digital property — phones imaged under warrants, drives seized in searches, communications intercepted under Title III, records pulled from providers. It sits in storage, lawfully held, for the cases it was gathered for. Then someone notices what it is: the richest imaginable pile of real-world training data — authentic messages, images, financial records, foreign-language chatter, the texture of actual human lives. Exactly what you need to build or fine-tune an AI model. So the government trains on it.
You were a bystander in one of those cases. You were never charged. You were never told your data had been collected, let alone that it went into a model. Years later, your photos and messages are distilled into the weights of a government AI used against strangers. What is your remedy?
Right now, practically none. Arguably, the antiquated Privacy Act – 5 U.S. Code § 552a could say something, but it is intended for government records, not third-party data. This post is about why § 10 must change that — and why training data is the use that proves the whole point.
Nothing in current law stops this, and that is the problem. Let me be careful: I am not saying any agency is doing it. I am saying the capability is here, the incentive is obvious, and the statute book is silent — which is exactly when a model act should get ahead of a practice rather than chase it. Every section I have drafted draws one line repeatedly: collection for one purpose does not authorize use for another. Section 7, the use restriction, says content gathered to investigate Case A may be used only for Case A’s authorized purposes. Training a general-purpose model is Purpose B at its most expansive — the content is conscripted to build a permanent tool aimed at everyone, indefinitely.
And training is not just another unauthorized use. It is the one with no undo. Every remedy upstream assumes the data can be handed back or destroyed: § 6’s retention clock, § 8’s disposition duty, an injunction to purge. Model weights break that assumption. Once your content has been absorbed into the parameters of a trained model, there is no meaningful delete key. You cannot un-train a model short of throwing it away and retraining from clean data, which no one volunteers to do. The disposition duty — the promise that the government will eventually get rid of what it should not keep — becomes a dead letter the instant the data is ingested. The content has escaped into a form none of the law’s existing tools can reach.
Worse, it can come back out. A well-documented failure mode of large models is memorization: under the right prompt, a model can emit chunks of its training data verbatim — a sentence, an image, a record. The claim that ‘it is only statistics, your file is not in there’ is empirically shaky. Your message, your photo, can resurface in the model’s output to a complete stranger who never had any lawful access to it. The content was not merely used; it was potentially re-disclosed — unpredictably, uncontrollably, to anyone who queries the model. That is disclosure by architecture, the § 7 problem I have written about, with the volume turned to maximum.
Now ask who this happens to, because that decides which remedy can even reach it. The people whose collected data trains a model are overwhelmingly not defendants in any case about that model. There is no prosecution, so there is no motion, so there is nothing to suppress. Suppression — last week’s remedy — is blind here. And, as I argued Tuesday and in Monday’s note on notice, these people are never told. You cannot bring an action to vindicate a use you have no idea occurred. The harm is, by design, invisible and unremedied: no charge, no notice, no suit.
Which is the whole case for § 10, and training data is its sharpest application. The model act needs a private right of action for any person whose collected digital property is used as training data in violation of the use restriction — available whether or not that person is ever prosecuted, and backed by the notice duty that lets them learn it happened at all. It needs a statutory damages floor, because the injury — your private life drafted into a government instrument — is real but impossible to price case by case. And it needs injunctive relief: an order to purge the data and, where it has already been ingested, to retrain or decommission the model built on it.That injunction is where the remedy bites hardest, and the reason is the same irreversibility that makes the harm so bad. Because un-training is so expensive, the credible threat of a court order to rebuild a model from clean data is enormous. It turns ingesting improperly repurposed data from a free lunch into a catastrophic liability — the kind that changes behavior before the fact. The permanence that makes the wrong uncurable afterward is exactly what makes the deterrent work beforehand.
You set the price high at the front end precisely because there is no fixing it at the back end.
Let me put the government’s best case as strongly as I can, because it is not frivolous. The data is already lawfully in hand; training on it could improve public safety — better translation of intercepted foreign-language communications, faster triage of lawfully seized evidence, detection tools that catch real offenders. Aggregate statistical learning, the argument runs, is not ‘disclosure’ of any individual’s content, because the model does not store your file as a file. And a damages-plus-injunction regime would chill beneficial government AI and hand every defendant a fresh way to get a model — and the cases built on it — thrown out.
Each point has a hard answer. ‘We already lawfully collected it’ proves far too much — taken seriously it would authorize any downstream use at all, which is the precise opposite of a use restriction; lawful collection for Purpose A has never been a license for Purpose B, and that principle is the spine of this entire statute. The ‘just statistics, nothing stored’ claim is false often enough to matter, because memorization and regurgitation are real — and even a model that never regurgitates has still conscripted your private life into a permanent government tool, without notice or consent. The harm is the use itself, not only the leak. And the chilling effect is not a bug; it is the calibrated point. The Act does not bar government AI. It bars training on content collected for another purpose. Train on data lawfully gathered for that purpose, or on synthetic or consented data, and you are clear. The remedy falls on one specific choice — repurposing seized private life as training fuel — which is the choice we want chilled.
Step back and this is the content-not-container thesis reaching its logical end. For ten weeks I have argued that the Fourth Amendment protects the content of your digital property in whatever form it takes — the original, the copy, the transcript. A model trained on your seized messages is your content in yet another form: a lossy, generative, irreversible representation of it, distilled into math. It is the ‘copy versus representation’ problem at its most extreme, because this representation can generate new text in the shape of your old life and cannot be recalled. If the content is protected, the protection must follow it, or it protects nothing.
That is the argument, and it is just this project’s first principle applied to its newest frontier: protection attaches to what the government does with your content, not to the container it arrived in. Training is the newest thing the government can do with it, and the only one with no undo. Suppression cannot reach it, because the people harmed are never charged. Disposition cannot cure it, because weights do not delete. Notice is what lets them find out, and a civil damages-and-injunction remedy — triggered the moment collected property is repurposed as training data — is the only thing that can make the government answer for it. Set the price before the bell rings, because once your life is in the model, you cannot take it back.
Next Tuesday, § 11: reporting and oversight — including the transparency that would at least force the government to say which data trained which model, so the question stops being invisible in the first place.
Leave a Reply