How Poisoning Rewrites the Odds at Deployment
For colluding adversaries in ML pipelines, train-test collusion is the case in which a training-time attack changes the model that a later deployment-time attacker will face. The key question is not whether poisoning lowers accuracy in the aggregate. It is whether poisoning changes memorization or behavioral distinctions in a direction that makes another attack more effective. That distinction lets the reader recognize the conditions under which training manipulation becomes a downstream enabler—and the conditions under which it does not.
Poisoning is often introduced as a way to make a model perform badly. That description is accurate for availability attacks, but it is too coarse for a pipeline-wide threat model. A poisoned model can fail broadly, fail on selected targets, respond to a trigger, or reveal sharper differences between records or groups. Those different changes leave different signals for later attackers.
Poisoning changes what the model learns
NIST defines poisoning as an attack during the training stage. In data poisoning, an adversary inserts or modifies training samples. In model poisoning, the adversary controls parameters or model updates directly. The objective may be an availability breakdown, where performance degrades across many inputs, or an integrity violation, where selected inputs are misclassified or routed to an attacker-chosen result. Backdoor poisoning is a particularly clear example: a model learns an association between a trigger and a target behavior, while ordinary inputs may continue to look normal.
The average test score may not reveal the most important consequence. A model can retain acceptable aggregate accuracy while learning rare examples too strongly, separating two populations too sharply, or responding unusually to records that resemble the training data. Such changes matter because many deployment-time attacks do not need the model to be generally inaccurate. They need the model to expose a reliable difference.
The SoK organizes the train-test bridge around three factors. They are not three universal measurements of model vulnerability. They are selected conditions that poisoning can influence and that later attacks can exploit.
Tail length describes how extended or imbalanced the training distribution is, especially toward rare or under-represented classes. When a distribution has a long tail, the model sees fewer examples for some classes than for the dominant head classes. The SoK connects poisoning and backdoors to extending that tail. Rare or unusual records may then be memorized more strongly, or may be learned less reliably, depending on the task and attack. Either outcome can create a useful difference for a downstream attacker: an unusual record may be easier to recognize as a member, while a rare class may be easier to evade.
Dataset distinguishability describes differences in model observables between records inside and outside the training set, or between models trained on datasets with different distributions. “Model observables” can include predictions, confidence scores, intermediate activations, parameters, or gradients. If those observables differ reliably, a later attacker has a signal with which to distinguish cases that should otherwise look similar. Poisoning can be optimized to increase such distinctions, according to the relationships gathered in the framework.
Subgroup distinguishability is the corresponding difference across demographic or other meaningful subgroups. A model may behave differently for people identified by race, sex, or another attribute. That difference can be a fairness problem in its own right, but it can also make an attribute-inference or distribution-inference attack more informative. A system that leaks a subgroup-related signal through its behavior can expose information even when no single prediction appears to reveal a sensitive attribute directly.
These factors connect training to deployment without requiring the attacker to insert a malicious instruction at query time. The model itself has become the carrier of the change. The second attacker interacts with the resulting behavior and extracts a signal that would have been weaker, noisier, or absent in the unpoisoned system.
One changed condition, several downstream risks
The same factor can support different objectives because the objectives ask related questions of the model. A membership-inference attacker asks whether a particular record was in the training set. Stronger differences between training and non-training records make that question easier. A distribution-inference attacker asks whether the model was trained on one population mix or another. Differences between models trained on those distributions provide the signal. An attribute-inference attacker asks for a sensitive property associated with a record; differences in behavior across subgroups can serve as evidence.
The SoK’s systematization therefore links poisoning or backdoors to evasion, membership inference, attribute inference, distribution inference, data reconstruction, and discriminatory behavior when the relevant arrows align. The mechanisms are not identical. Tail behavior can expose under-represented classes to evasion or make rare records more memorable. Dataset distinguishability can help identify membership or a global property of the training population. Subgroup distinguishability can make attribute inference and unequal treatment more pronounced. Data reconstruction can benefit when the model preserves distinctive information about records or subgroups.
The important detail is what the later attacker is trying to distinguish. An evasion attacker needs a region of input space where a small or feasible change can move the prediction. A membership attacker needs a difference between how the model treats a record it has seen and one it has not. A distribution attacker needs a signal that survives aggregation across many records and separates competing descriptions of the training population. An attribute attacker needs behavior that correlates with a sensitive feature, while a fairness assessment compares that behavior across groups. These objectives may all touch the same model, but they do not consume the same evidence.
That difference explains why the framework keeps the factors separate. A longer tail may create more memorable outliers, yet it may also leave the rare class poorly represented. Distinguishability across datasets can strengthen a population-level inference without making any one record reconstructable. Subgroup distinguishability can expose a sensitive pattern while the model remains accurate on the overall test set. The downstream edge is therefore a claim about a particular signal and objective, not a general statement that poisoning has made the model uniformly more vulnerable.
This is why a single “model quality” number is a poor proxy for cross-stage risk. Accuracy averages over examples. Privacy attacks often focus on individual records or population properties. Fairness attacks compare groups. Evasion attacks look for inputs that cross a decision boundary. The relevant evidence may be concentrated in a small region of the data, precisely where an aggregate metric has the least resolution.
The relationship can also run through a change that looks beneficial under one objective. Suppose poisoning makes a model less reliable on a rare class. That may make the class more vulnerable to evasion, while simultaneously making a surrogate model less able to reproduce the target’s behavior on that class. The same tail extension has different consequences because the downstream attacks require different properties.
The diagram shows a family of possible relationships, not a promise that one poisoned model will amplify every listed risk. The factors can interact, and the exact effect depends on the model, data, attack objective, and access assumptions. It is more accurate to say that poisoning creates candidate dependency edges that require validation.
Why a poisoned model may resist extraction
The model-extraction case is a useful boundary because it runs against the intuition that making a target more unusual must make it easier to steal. In the SoK’s framework, the relevant factor is tail length. Extending the tail can cause a target model to learn under-represented records insufficiently. An attacker who trains a surrogate from the target’s outputs then has a harder learning problem: the target’s behavior on the rare portion of the task is less coherent or less faithfully captured. The proposed arrows therefore oppose one another. Poisoning increases the tail, while model extraction is associated with a decrease in the surrogate’s utility or fidelity under that condition.
The paper tests this conjecture on CIFAR10 and CIFAR100. It uses a ResNet target, a VGG16 surrogate, backdoor poisoning, and a knockoff-style extraction procedure. Across the tested poisoning levels and query budgets, the reported surrogate accuracy and fidelity are lower than the no-poison baseline for the comparison. The target itself is still evaluated, so the result is not simply that the poisoning destroyed the whole experiment. It supports the narrower interpretation that, in this setup, the poisoned target is not a better extraction target.
That result illustrates the alignment test. To claim useful collusion, it is not enough to point out that poisoning and extraction both concern the same model. The first attack must produce a factor that the second attack benefits from. If poisoning makes target behavior less learnable for a surrogate, then the two adversaries have no reinforcing edge through that factor. They may still have other relationships under other conditions, but this one does not establish them.
The distinction between “no useful collusion through this factor” and “no interaction of any kind” is important. A different poisoning objective might alter architecture, expose a trigger, or change the query responses in a way that affects extraction. A different extraction method might benefit from a distinctive behavior rather than suffer from poor generalization. The supplied experiment does not settle those possibilities. It demonstrates why the direction of a relationship has to be checked rather than assumed.
Correlation is not a causal law
The SoK presents its guideline as a way to conjecture about collusion by matching common factors and checking whether the arrows align. That is a practical starting point, but it is not a causal model of how every training process behaves. ML training is affected by data composition, optimization, architecture, regularization, evaluation choices, and randomness. A factor may correlate with attack effectiveness without being the only mechanism responsible for it.
The paper also deliberately selects some factors and omits others. Hyperparameters such as model capacity or training-set size are not treated as freely controllable by many poisoning roles, and some possible influences are considered insufficiently explored. This keeps the framework tied to attack assumptions that can be discussed, but it also limits the completeness of the explanation.
NIST’s taxonomy reinforces the need for that caution. It distinguishes data poisoning, model poisoning, targeted poisoning, backdoors, and clean-label variants because their capabilities and effects differ. A data contributor who cannot control labels faces a different problem from an actor who controls model updates. A backdoor may require control over both training and testing data, while a model-poisoning attack may operate through parameters or federated updates. “Poisoning” is therefore a family name, not a single intervention with a fixed downstream effect.
Defenders should consequently measure more than clean accuracy after a suspected training-time attack. They need to ask whether the model now separates training members from non-members more sharply, whether rare classes or subgroups behave differently, whether the model leaks a distributional property, and whether a surrogate can reproduce its behavior. Those are not universal test requirements for every system; they are examples of checking the factors that connect the known attack to plausible downstream objectives.
The train-test lesson
Train-test collusion changes the meaning of training integrity. Protecting the training set is not only about preserving the model’s immediate utility. It is also about preventing an upstream participant from shaping the signals that a later attacker will exploit. Filtering suspicious samples, validating provenance, and inspecting model behavior can therefore have effects beyond the attack they were designed to address.
But the lesson is not that every poisoned model becomes a universal amplifier. The downstream effect depends on which factor changed, in which direction, and whether the later attacker can use that factor under their own access and knowledge constraints. The paper’s positive relationships and its negative model-extraction case belong together: aligned arrows suggest a path; opposing arrows close one path; neither is a proof of the whole threat landscape.
The result is a more precise question for the next stage of analysis. If a deployed model can be queried, what information or data can one test-time attack produce for another? That moves the focus from a changed model condition to an exchanged artifact—and from poisoning the pipeline to learning from it.