Large Language Model Tokenizers: Privacy and Membership Inference

Overview

This series explains how large language model tokenizers turn corpus frequencies into a public vocabulary and merge order that can retain dataset-level membership signals. It follows the path from BPE compression through shadow-tokenizer attacks and controlled C4 website-level evidence, then compares min-count filtering with randomized merge selection. The series ends with NIST’s boundary for a meaningful differential-privacy claim: define the protected unit, account for ε, δ, and composition, measure utility and bias, and secure the full release system. The evidence does not by itself identify a person’s document or prove that tokenizer and downstream-model corpora match.

Learning outcomes

A public LLM tokenizer can support dataset-level membership inference through learned vocabulary and merge structure.
The reported evidence concerns website-level datasets; it does not prove an individual document entered a downstream model.
In tested settings, larger vocabularies and candidate datasets strengthened the signal without making every dataset inferable.
Min-count filtering reduced tested attacks with utility costs, while randomized training targets a stronger but demanding privacy framework.
A meaningful DP claim must define the privacy unit, account for ε, δ, and composition, and protect the whole release system.

Keywords

LLMTokenizerPrivacyMembership inferenceDifferential privacyBPEVocabularyMerge orderCompressionUtility

Chapters

How LLM Tokenizers Can Reveal Training Data

A learned vocabulary carries traces without proving individual use

11 min · A learned vocabulary carries traces without proving individual use

11 min

How LLM Tokenizers Turn Frequency into Vocabulary

BPE turns frequency competition into an observable merge history

14 min · BPE turns frequency competition into an observable merge history

14 min

Testing Corpus Membership with LLM Tokenizers

Why with-candidate and without-candidate shadows matter

12 min · Why with-candidate and without-candidate shadows matter

12 min

LLM Tokenizers: Evidence and Limits

What website-level experiments can and cannot establish

13 min · What website-level experiments can and cannot establish

13 min

LLM Tokenizers: Defenses and Utility Costs

Comparing min-count filtering with randomized training

12 min · Comparing min-count filtering with randomized training

12 min

LLM Tokenizers: Defining a Real Privacy Guarantee

Why ε, δ, composition, and the privacy unit all matter

14 min · Why ε, δ, composition, and the privacy unit all matter

14 min

LLM Tokenizers: The Whole-System Privacy Boundary

Connect the corpus, artifact, attack, defense, and deployment

20 min · Connect the corpus, artifact, attack, defense, and deployment

20 min

Preview every chapter · Unlock full text, audio, and video from chapter 4

Plans

Ch.1-3 FreeCh.4-7 Premium

FAQ

What can a public LLM tokenizer reveal about its training data?

It can reveal whether a candidate dataset is compatible with the tokenizer’s training history at the dataset level. That does not identify a person or prove the downstream model used the same corpus.

Does tokenizer membership inference prove that a person’s document trained an LLM?

No. The reported attacks classify website-level datasets and do not by themselves prove that a person’s document was used or that the downstream LLM shares the tokenizer’s corpus.

Why do vocabulary size and dataset size affect the reported signal?

Larger vocabularies create more room for distinctive patterns, and larger candidate datasets can supply more such evidence. The reported trend is conditional on content, duplication, corpus mixture, and tested ranges.

Do defenses that lower attack accuracy guarantee tokenizer privacy?

No. Min-count filtering is empirical mitigation, while randomized training can support a DP analysis only when the privacy unit, parameters, composition, implementation, and release boundary are defined and checked.