The Privacy Boundary Is the Whole System

This final synthesis answers the opening question: what does it mean for a tokenizer to leak the training set? The evidence supports a sharper conclusion than either alarm or reassurance. A public tokenizer can preserve enough training-dependent structure for dataset-level membership inference, but that signal is not the same as proof of an individual’s document or a downstream model’s memorization. The right judgment connects the corpus, the tokenizer’s learning path, the public artifact, the attack, the defense, and the privacy guarantee as one system.

The causal chain is the finding

The chain begins before an LLM generates anything. A corpus supplies text to tokenizer training. In the BPE setting studied by Tong and colleagues, the trainer repeatedly merges frequent symbol pairs into a vocabulary and records the order of those merges. The result is useful because frequent patterns can be represented compactly. It is also data-dependent because the frequency landscape—and the competition among candidate merges—comes from the corpus.

The next link is release. Tokenizers are exposed so users and applications can reproduce token counts and understand billing or input limits. That public interface provides a vocabulary and merge structure without publishing the raw corpus. The release is lossy: it does not preserve every count, discarded candidate, or preprocessing decision. But lossiness does not mean irrelevance. A compact representation can retain enough information to distinguish some training histories.

The attack supplies the counterfactual. An adversary takes a candidate dataset, obtains auxiliary data from a related distribution, and trains shadow tokenizers with and without the candidate. Global merge-order similarity is a weak baseline in the study because ordinary web patterns make the broad distributions look alike. The stronger methods focus on distinctive tokens or estimate whether the candidate was necessary for some token to appear. The target is then classified according to which shadow population it resembles more.

This sequence explains why the privacy surface is easy to miss. Each local step has a legitimate purpose. Compression makes the model’s input manageable. A public tokenizer makes token billing transparent. Shadow training creates a controlled way to measure risk. Yet the combination produces a public artifact that can answer a question about its own origin. Privacy exposure emerges at the interface between steps rather than from a single obviously dangerous feature.

The evaluation makes that chain measurable. In the controlled C4 experiment, the researchers treat each of 4,133 websites as a dataset, use 1,681,296 pages, train target tokenizers on half of the datasets, and vary the vocabulary size from 80,000 to 200,000 tokens. A bag-of-words baseline has AUC 0.513, close to chance, which supports the interpretation that the main attacks are not simply recognizing a content-distribution difference in the candidate pages. Vocabulary overlap reaches AUC 0.771 and frequency estimation 0.740 at 200,000 tokens in the reported setup.

The scale relationship strengthens the warning. Both principal attacks perform better for larger candidate datasets in the tested ranges. At the largest vocabulary size, vocabulary overlap reaches AUC 0.882 for candidate datasets with 800 to 1,200 samples, and frequency estimation reaches 0.843. The paper also reports meaningful true-positive rates at low false-positive rates. These results do not say that every dataset is inferable. They say that a collection with enough distinctive evidence can become more distinguishable as the tokenizer’s capacity and the candidate’s size increase.

The important connection is between capacity and evidence, not between size and guilt. A larger vocabulary gives more candidate strings a chance to survive as tokens, while a larger dataset supplies more opportunities for a rare pattern to become frequent enough to survive. When both grow together, the public artifact can preserve more of the candidate’s statistical signature. But the signature still depends on what the candidate contains and what else appears in the corpus. A large, repetitive collection can be less distinctive than a smaller collection containing unusual names or code identifiers; a large vocabulary can also be filled mostly by patterns shared across many sources.

This gives the result two different uses. As a research finding, the scale trend warns model and tokenizer designers that capability improvements can enlarge a privacy surface they were not measuring. As an audit finding, it suggests that candidate-dataset size and vocabulary size should be recorded whenever an attack score is reported. As a governance finding, it means that a company cannot infer safety from the fact that its tokenizer is only a component or that its public interface contains no raw passages. The component’s size and training role are part of the risk calculation.

The causal chain also explains why the evidence cannot be reduced to a dispute about whether tokens are “memorized.” The attacks need not show verbatim reproduction.