From Next-Token Prediction to a Membership Question
For the Window-Based Comparison attack, the crucial object is not a model’s final answer but the loss it assigns to each token in a candidate record. This chapter makes that signal concrete: readers will see how next-token prediction becomes a sequence of local observations, why a fine-tuned target is compared with a pre-trained reference, and where the score-based access boundary limits the conclusion.
A language model scores one token at a time
An autoregressive language model reads text from left to right. At each position, it uses the tokens already seen—the prefix—to assign probabilities to what might come next. A token is the unit the model actually processes. It may be a whole word, part of a word, punctuation, or a short sequence of characters, depending on the model’s tokenizer.
Suppose a candidate sequence is represented as x₁, x₂, …, xₙ. The model does not treat the sequence as one undifferentiated object while it predicts it. It estimates the probability of x₁, then x₂ given x₁, then x₃ given x₁ and x₂, and so on. The probability of the complete sequence is built from those conditional predictions. The important consequence is that the model produces a separate judgment at each position.
To express that judgment in a convenient direction, researchers use per-token loss, or negative log-likelihood. For model M at position i, the loss is the negative logarithm of the probability assigned to the observed token given its prefix. A token the model considers likely receives a lower loss. A token it finds surprising receives a higher loss. Loss is therefore a numerical record of the model’s uncertainty as it moves through the sequence.
That definition is simple, but it changes what can be observed. A sequence-level average compresses all of those local surprises into one number. A per-token loss sequence retains their order and location. Two candidate records can have the same average loss while differing sharply in where the model is confident, uncertain, or unusually changed. For a membership test, that difference in shape may matter more than the final average.
Consider two candidate sequences that produce the same overall amount of target-versus-reference improvement. In the first, the improvement is spread across many modest positions. In the second, nearly all of it comes from one unusual token, while the rest of the sequence shows little change. A global summary sees similar totals. A position-by-position record preserves the reason the totals arose. The first pattern could be compatible with repeated, distributed learning; the second could be a domain-specific outlier. Neither pattern proves membership, but they should not be treated as equivalent evidence.
The ordering also matters because each loss is conditional on a prefix. The model is not scoring a bag of words in which positions can be freely rearranged. The same token can receive a different loss when its preceding context changes. A meaningful comparison therefore presents the same candidate sequence to both models and keeps the positions aligned. If the evaluator changes the prompt, normalizes the text differently, or compares scores produced by incompatible tokenizations, a difference may reflect the evaluation setup rather than the fine-tuning update. The paper’s reference-model experiments later illustrate why comparability is a real calibration concern.
This local view is not a claim that every token has an independent meaning. Neighboring predictions share context, and the model’s update can affect several positions at once. It is a claim about preserving observable structure long enough to ask a better question. Before an evaluator decides how to combine the positions, they need to know whether a large score came from a stable pattern, a single extreme event, or a mixture of both.
The membership question itself is precise. A member is a record that was included in the dataset used for fine-tuning. A non-member is a record held out from that training process, ideally drawn from a comparable distribution. The evaluator is not asking whether the candidate is about the same topic as the training data, whether the model can produce a similar sentence, or whether the record resembles something the model knew before fine-tuning. The question is whether this particular record crossed the boundary into the fine-tuning set.
That boundary is often hidden from an outside observer. The evaluator can submit a candidate sequence to a model and inspect how the model scores it, but they do not receive a label saying “seen” or “unseen.” The score is indirect evidence. It becomes useful only if the model’s behavior differs in a systematic way between members and non-members.
Fine-tuning supplies a reason such a difference might exist. A pretrained model has already learned broad language patterns from earlier data. Fine-tuning then updates its parameters on a smaller, more specialized collection of examples. The update can improve performance on the new domain, but it can also make some training examples unusually familiar. The model may assign lower loss to a member than to an unseen record, not because every token in the member is memorized, but because the update has changed the model’s response to parts of that sequence.
A useful way to think about the score is as a trace of changed behavior, not as a direct window into the model’s memory. If a target assigns low loss to a phrase, that could mean the phrase is common in the domain, that the target has learned a general pattern, or that this particular example influenced the update. The loss value by itself does not identify which explanation is correct. Membership inference depends on comparing the target’s change with a suitable baseline and then aggregating evidence across many examples.
This is why the sequence matters. A candidate record does not carry one privacy signal that is evenly painted across every token. It presents a series of opportunities for the fine-tuned model to behave differently. Some positions may be unaffected. Others may reflect broadly useful learning. Still others may contain a small trace associated with the example’s inclusion. Preserving the per-token sequence keeps those possibilities separate long enough to analyze them.
The difference needs a reference—and an access boundary
The paper’s attack uses two models. The target model, written as Mᵀ, is the fine-tuned system whose privacy is being assessed. The reference model, Mᴿ, is preferably the pretrained base from which the target was adapted. The reference has not been exposed to the fine-tuning data, so it provides a comparison point for what the candidate looked like before that update.
For the same token position, let the target produce loss ℓᵀᵢ and the reference produce loss ℓᴿᵢ. WBC defines the per-token difference as:
Δᵢ = ℓᴿᵢ − ℓᵀᵢ
The order of the subtraction matters. A positive Δᵢ means the reference is more surprised than the target at that position: the fine-tuned target assigns the token a lower loss and is more confident relative to the reference. A value near zero means the fine-tuning changed the two models’ scores little there. A negative value means the target has higher loss than the reference at that position.
This sign is not a membership label. It is one observation whose meaning depends on the surrounding evidence. A member may contain many positions where the target is more confident, but it does not follow that every positive difference is membership-specific. A domain term that appears throughout the fine-tuning corpus can make the target more confident for members and non-members alike. Conversely, the paper reports that the most informative membership shift can appear where the target is slightly less confident, in the left tail of the difference distribution. The direction of one token is therefore not enough; the sequence of directions is what later aggregation will examine.
The reference model removes one major source of ambiguity: the target’s general ability. Imagine that the target assigns a low loss to a candidate simply because it is a stronger language model than the reference. That absolute score would say little about whether the candidate was in the fine-tuning data. If the same candidate is scored by the base model, the evaluator can ask a more specific question: how much did the fine-tuning change the target’s treatment of this text relative to its starting point?
The comparison is most informative when the reference is genuinely related to the target. The paper calls the original pretrained base the most principled choice because it isolates the effect of fine-tuning. But it also tests references with different sizes and architectures. Those mismatches do not make every comparison useless, yet they can change the result. A different tokenizer or model family may introduce its own score differences, making it harder to tell which changes belong to fine-tuning and which belong to the comparison baseline. The reference is a calibration instrument, not a neutral oracle that can be swapped without consequence.
The paper’s threat model is also narrower than the phrase “black-box attack” might suggest. The evaluator does not need the target’s parameters, gradients, or hidden-state activations. They do need score-based access: the target must return a loss value for each token in the submitted sequence. The evaluator is assumed to have equivalent score access to the reference model as well.
That distinction determines whether the reported procedure can be used. An open-weight model can be run locally, so per-token scoring may be available to anyone with the weights and the necessary inference setup. Some hosted inference systems also expose token-level log probabilities or prompt log probabilities. In those cases, an evaluator may be able to obtain the required sequence of scores. But a service that returns only generated text, a class label, or one average loss for the entire prompt does not expose the same observation. It may support a different privacy test, but it does not provide the input required by WBC as described in the paper.
The diagram also highlights an important control boundary: the evaluator supplies the candidate sequence, while each model supplies only its score sequence. The comparator owns the arithmetic, not the model’s internal state. This makes the method less demanding than a white-box analysis, but it does not make it universally available. Access to per-token scores is still a meaningful capability, and access to a reference model is an additional dependency.
It is equally important to keep the output claim bounded. WBC produces a real-valued membership score, which can then be compared with a threshold or used to rank candidates. A high score means the candidate resembles the member behavior learned by the evaluator’s attack under its chosen models and data. It does not, by itself, prove that the target stored a verbatim copy, reveal which training step caused the change, or establish that a person’s information was exposed in full. Those are different questions, requiring different evidence.
The comparison also does not require the target to be bad at generalization. A target can become better on both member and non-member examples while still treating some members differently enough to create a detectable pattern. Nor does a reference comparison guarantee that every detected difference is personal memorization. It only creates the contrast needed to investigate that possibility. The distribution of differences must still be examined to separate common domain adaptation from sparse membership evidence.
That is the reason to preserve the entire difference sequence instead of immediately averaging it. A single number can hide whether its magnitude came from many modest changes or one extreme event. Once the target and reference have produced their local scores, the next question is not how to calculate another average. It is which kinds of local change carry information about inclusion, and which merely reflect what the fine-tuned model learned about the domain as a whole.