The Residue of Meaning: Why Removing Names Is Not Enough
The problem in this chapter is narrower than “can AI invade privacy?” It is how a pseudonymous text profile becomes linkable when names and handles are gone but meaning, relationships, and repetition remain. The practical judgment is concrete: when assessing a corpus or account, look for the combinations and connections that make a record distinctive, not only for strings that look like personal identifiers.
From Direct Identifiers to Distinctive Profiles
A direct identifier is information that points to a person without much interpretation: a name, phone number, personal URL, or account handle. Removing such fields is often necessary. It is not sufficient, because identity can also be inferred through attributes that point indirectly. NIST calls these quasi-identifiers, or indirect identifiers. They may describe many people individually, yet narrow the possibilities when combined with other information.
In a spreadsheet, a quasi-identifier might be a birth year combined with a state. In a narrative, it may be a cluster of references spread across several sentences: a profession, a location, a specialized interest, a distinctive life event, and a time marker. None needs to be unique by itself. The combination can act like a fingerprint when an outside party has another source with overlapping facts.
The USENIX study uses the term micro-data for observations at the level of an individual. Its examples include a movie rating, a location, an occupation, or a writing habit. The point is not that one preference names someone. The point is that a set of preferences or habits can be compared with a second set that is connected to known identities. The old Netflix Prize deanonymization attack illustrated this with sparse movie ratings: a collection of apparently ordinary choices became identifying when matched with public profiles.
Text makes the same mechanism less tidy and, in some ways, more expressive. A post can reveal what a person knows, what they care about, where they have been, how they describe a problem, or which events they experienced in a particular order. These are not necessarily “fields” that a redaction tool can find. They are relations among facts. A model that reads for meaning can turn them into a structured impression even if the original text contains no line saying “the author is this person.”
That does not make every detail equally dangerous. A common interest shared by millions may add little identifying power. An institution may be too broad to distinguish one individual. A local business may identify a place but not its patrons. The risk comes from the joint pattern and from the availability of an outside dataset that can supply the missing link. The same detail can be harmless in one population and highly informative in another.
High dimensionality intensifies this effect. A profile with many retained attributes has more possible combinations than a profile with only one or two. Even if each dimension is ordinary, the intersection may be rare. NIST warns that data can remain identifying after unique identifiers are removed and quasi-identifiers are manipulated when the record is sufficiently high-dimensional and can be linked to an identity. In this setting, “high-dimensional” does not require exotic mathematics. It means that the record preserves enough separate facts, relationships, or sequence information to distinguish it from its neighbors.
Consider two pseudonymous writers who both discuss machine learning. That broad topic says little. One writer also mentions a particular research subfield, a move from one region to another, a niche technical frustration, and a recurring reference to a local activity. The combination may reduce the candidate set sharply if a public professional page, conference paper, or older post contains overlapping facts. The example is conceptual, not a claim that these clues always suffice. Its purpose is to show why privacy analysis must consider intersections rather than classify each sentence as safe or unsafe in isolation.
A narrower candidate set is not the same as a verified identity. The outside source may be incomplete, stale, or about a different person with a similar profile. A matcher can find a plausible overlap and still make a false attribution. The reverse can happen too: the right person may be present in the outside source but the relevant clue may be missing, generalized, or expressed differently. This distinction matters for both privacy and fairness. A system that abstains often may identify only the most information-rich users, while a system that guesses too readily may attach sensitive statements to the wrong people. Linkability is therefore a property of the data and available sources; correctness is a property of a particular matching decision.
The source of the auxiliary information also changes the risk. A public professional biography may make one kind of connection possible, while a sequence of posts or a restricted dataset may support another. The same pseudonymous profile can be difficult to match for a casual reader and easier to match for an organization that already holds a related record. NIST’s language about an anticipated recipient is useful here: the question is not whether an abstract attacker could imagine a connection, but what information and resources a plausible recipient can actually use. That keeps the analysis grounded without pretending that the recipient’s knowledge is fixed forever.
The same reasoning applies to attribute inference even when identity is not recovered. A collection of text may make a location, occupation, health-related concern, or political interest easier to infer. NIST treats disclosure broadly: a release can reveal identity, attribute information, or improve an observer’s inference about a person. A person does not need to be publicly named for the release to change what others can know about them.
The flow is not deterministic. Each arrow represents an opportunity for information to become more discriminating, not a guarantee of success. If the profile contains little information, if no suitable auxiliary data exist, or if the transformation destroys the relevant relationships, the chain may break. A defensible assessment asks where it breaks for the people and adversaries that matter, rather than assuming that it breaks because a name was deleted.
The dimensions of a profile are not interchangeable. A dozen broad interests may be less distinctive than two details tied to an unusual time and place. A relationship between facts can carry more information than either fact alone: a workplace connected to a specialized project, or a recurring activity connected to a particular schedule. Sequence matters as well. A statement made once may be ordinary; the order in which several events appear can reflect a life history. Even absence can be informative in a carefully bounded dataset, although such inferences are uncertain and should not be treated as proof.
This is one reason semantic sanitization is harder than deleting a vocabulary of sensitive words. A tool may remove an explicit name while preserving the sentence that explains how the writer knows a particular person, or it may replace a location while leaving a timeline intact. A reviewer may see each remaining sentence as harmless and miss the pattern that emerges when the sentences are read together. The relevant unit is not always the token or line. It can be the profile, the sequence, or the relationship among records.
Linkage Outlives a One-Time Release
Pseudonyms create a useful tension. They can separate a public contribution from an offline identity, but they can also create continuity. If the same pseudonym appears over months, observations that would be unrelated in separate one-off releases can be assembled into a longitudinal profile. Continuity helps a community recognize a returning participant and helps a data owner connect records for legitimate purposes. It also gives a matcher more evidence about stable interests, habits, and circumstances.
NIST describes several kinds of linkage beyond an explicit username. Family identifiers, distinctive names, device identifiers, and geographic locations can connect records. Dependencies between records can do the same even when no field was intended to be a join key. A sequence of measurements may form a distinctive constellation. A sequence of locations may reveal a behavioral pattern. In text, the analogous structure can be a recurring mix of subjects, time references, relationships, and ways of describing experience.
This is why changing a direct identifier to a stable replacement is not the same as removing linkability. A repeatable pseudonym lets multiple observations continue to refer to the same subject. If a key or transformation is exposed, the original association may be compromised. Even if the original identity is not recovered, stable replacement can make it easier to match records across releases or organizations. NIST warns that encryption and hashing used to protect direct identifiers depend on careful key management and can create broad consequences when procedures or keys fail. A plain hash without a secret key may also be vulnerable to guessing because the space of possible input values can be searched.
Aggregation has a related weakness. Summarizing a dataset can reduce the precision of any one observation, but a series of summaries can reveal changes. NIST gives a simple example: if a school publishes grouped counts at two times and one new student arrives between them, the difference may reveal something about that student’s performance. The example is not about schools in particular. It demonstrates that a release can disclose information through comparison with an earlier release even when neither table appears revealing alone.
Composition is the broader name for this problem. A protected text release may later be combined with another release, a newly public profile, a data breach, or a dataset created by a different organization. The future source may not exist when the original decision is made. A transformation that seemed adequate against the information available at publication can become inadequate as the outside world changes. NIST therefore treats post-release monitoring and periodic reconsideration as part of responsible data handling, not as optional cleanup.
Time changes both sides of the equation. People’s interests and circumstances evolve, which may make a temporal match harder in some cases; stable characteristics and repeated habits may make it easier in others. External datasets also accumulate. A later publication can supply a missing employer, location, or date, while a platform change can expose a new connection between accounts. The original release does not become more identifying because the text itself changed. It becomes more identifying because the surrounding information changed.
This creates an important asymmetry between publication and correction. A curator can remove a file from an official site, but copies, quotations, indexes, or downstream analyses may remain. A public release can therefore outlive the assumptions and policies under which it was approved. Controlled access does not eliminate risk, but it can limit who receives the raw semantic material and make later review more feasible. The choice between public release and restricted access is part of the identity-risk model, not an administrative detail after the technical work is finished.
There is a limit to what this conceptual model should become. It would be possible to describe a matching process in enough operational detail to help someone pursue a real person. That is not necessary to understand the defensive issue. The useful abstraction is to identify the interfaces where risk grows: text is converted into features, features persist across contexts, an auxiliary source supplies candidates, and repeated releases add evidence. The model supports threat assessment without naming individuals or reproducing a procedure for targeting them.
The main lesson is therefore more precise than “redaction fails.” Some transformations work for some goals. Removing a personal URL may stop an immediate lookup. Generalizing a location may reduce precision. Restricting access may make a rich corpus available to a qualified group without making it searchable by everyone. But the effectiveness of each choice depends on the data, the intended use, the auxiliary information, and the consequences of error. NIST explicitly warns against treating a familiar transformation as safe without evaluating it against the actual threat model.
A text corpus should be judged as a whole system of signals and relationships. Ask what facts survive, whether they can be linked across time or platforms, what other releases exist, and what new sources might appear. Ask also what utility would be lost by removing context. Privacy and usefulness are not independent dials: reducing detail can protect people while also making the corpus less valuable, and retaining detail can preserve the very structure that makes analysis possible.
A useful threat model makes those questions explicit. Who is the recipient: the general public, a vetted researcher, a platform operator, or an organization with a separate customer database? Is the goal to identify a named person, to find any match in a population, or merely to infer a sensitive attribute? Does the recipient receive one snapshot or repeated updates? What harm follows from a wrong match, a correct identity disclosure, or a more accurate inference? These distinctions determine which signals need protection and which controls are proportionate. They also prevent a privacy review from treating “the internet” as one uniform adversary with one fixed capability.
That gives the LLM problem its proper starting point. The model is not the original source of identity. The corpus, the surrounding public information, and the continuity of the traces create the opportunity. LLMs matter because they can organize this distributed residue into a repeatable matching workflow. The next chapter examines that workflow as a system: what extraction, search, reasoning, and calibration each contribute, and why a plausible similarity score is not the same thing as a verified identity.