Why the First Fixes Cost Utility
The tokenizer attacks expose a specific signal: infrequent, distinctive patterns can become stable enough in a vocabulary to distinguish a candidate dataset from a non-member. This chapter compares two responses to that signal. One removes low-count tokens after training; the other randomizes the merge decision during training. The comparison helps readers judge what each defense changes, what it costs in compression utility, and why lower attack accuracy is not automatically a formal privacy guarantee.
Filtering the output after training
The min-count defense starts from the trained vocabulary and removes tokens whose aggregate training count falls below a chosen threshold. Its target is direct: the rare entries most likely to be distinctive to one dataset are also the entries most likely to carry a membership signal. If those entries are absent from the released vocabulary, an attacker has fewer token-level differences to compare between the target and the IN/OUT shadow populations.
The mechanism is easy to understand because it changes the artifact at the point where it is published. Imagine that a token appears only a handful of times in the full tokenizer corpus and most of those occurrences come from one candidate website. The token may be valuable evidence for vocabulary overlap, but it may contribute little to broad language compression. A min-count threshold can remove it without trying to identify which website caused it. The defense treats low frequency as a risk indicator, not as a legal or semantic judgment about the token’s source.
That simplicity is also the limit. Frequency is only a proxy for distinctiveness. Some low-count tokens may be useful for code, names, multilingual text, or specialized terminology. Some high-count tokens may still be concentrated in a candidate collection or reveal a pattern through their merge position. Removing tokens below a threshold therefore reduces one class of signal without proving that the remaining vocabulary is independent of the training data.
Choosing the threshold is consequently a workload decision, not just a privacy setting. A low threshold preserves more of the vocabulary and removes only the most obviously rare entries. That may protect general utility while leaving enough distinctive tokens for a large candidate dataset to be recognized. A high threshold removes more potential signals, but it also discards more domain-specific vocabulary. If the tokenizer serves code, medical terms, or multiple languages, the entries that look rare in the aggregate corpus may be exactly the entries users need most.
The threshold can also shift the burden rather than remove it. When a specialized token disappears, the same text is represented by a sequence of smaller tokens. Those pieces may be more common and less revealing individually, but the longer sequence can affect token limits and billing. It can also change how a downstream model receives a domain-specific phrase. The paper measures the resulting compression change, but bytes per token is only one view of the operational cost. A defense decision should ask which text is affected, not only whether the average benchmark score moved.
Most importantly, the filter has no way to know why an entry is rare. It does not distinguish a harmless idiosyncrasy from a sensitive identifier, a valuable technical term, or a string whose presence would reveal a training relationship. It applies the same rule to all low-count entries. That uniformity makes the mechanism simple to explain and audit, but it prevents the threshold from being a precise privacy classifier.
The paper’s results show partial reduction rather than elimination. For a 200,000-token tokenizer, vocabulary overlap reaches an AUC of 0.771 without the min-count defense.