The Attack Is Not One Attack
This series examines colluding adversaries in machine-learning pipelines: people or organizations with different positions, goals, or capabilities that coordinate so one attack makes another more effective. The immediate focus is the blind spot created by treating poisoning, evasion, privacy attacks, model theft, and fairness failures as separate items. The practical judgment it supports is simple but consequential: when assessing an ML system, ask not only what each attacker can do alone, but also what one attacker can change, learn, or hand to the next.
That question matters because an ML system is not a single program receiving a single hostile input. It is a chain of data collection, code and model supply, training, evaluation, deployment, and interaction. Each stage leaves something for the next stage to consume. A training set becomes a model; a model becomes an API; API responses become observations, auxiliary data, or clues about the system. If a threat model examines each transition in isolation, it can miss the way risk accumulates across the chain.
The blind spot is in the connection
A conventional security assessment might ask whether a model is vulnerable to an evasion attack, whether its training data is exposed to membership inference, or whether its API can be copied through model extraction. Those are useful questions. They become incomplete when their answers are treated as independent.
Consider a black-box service that returns predictions but does not disclose its architecture. One attacker may send carefully chosen inputs and retain the resulting input-response pairs. The attacker may be interested in evading the service, while another party wants data with which to build a shadow model. The first activity has produced an asset for the second. The parties might sell the information, share the cost of querying the service, or simply coordinate their efforts. The important point is not the business arrangement. It is that the first attack changes the second attacker’s starting conditions.
The supplied SoK, or systematic knowledge paper, uses “collusion” in this specific sense: explicit coordination in which one adversary executes an attack and uses its outcome to improve another attack. This is narrower than saying that a system has several vulnerabilities. Two attacks can coexist without helping one another. Collusion requires a useful connection between them—an altered model condition, a dataset, a distribution estimate, or knowledge about model behavior that satisfies a later prerequisite.
That distinction changes what counts as evidence. A list of risks tells a defender what might go wrong in separate scenarios. A collusion analysis asks whether the output of one scenario is an input to another. It is a shift from a catalogue to a dependency model.
The diagram is deliberately modest. It does not say that every outcome flows to every later attack. It shows why the question can be asked at all: the same model lifecycle contains transitions through which conditions and information can travel.
The pipeline creates the opportunity
The NIST taxonomy of adversarial machine learning provides a useful vocabulary for seeing those transitions. It separates predictive-AI attacks by lifecycle stage, attacker objective, capability, and knowledge. Poisoning occurs during training, when an attacker inserts or modifies data or controls model parameters. Evasion and many privacy attacks occur after deployment, when an attacker modifies inputs or queries a trained model. The distinction helps locate an attack, but it does not imply that the stages are sealed off from one another.
The handoffs are also handoffs of authority. A data provider may determine which records enter training. A code or model provider may supply components that a trainer treats as trustworthy. A model owner may pass the trained model to a service provider, who exposes an interface to clients. In a smaller organization, several of these roles may be held by the same team; in a larger supply chain, they may be separated across vendors and institutions. Either way, the security question is about who can alter state or observe it at each boundary.
NIST describes capabilities such as training-data control, model control, testing-data control, source-code control, and query access. It also distinguishes white-box, gray-box, and black-box knowledge. These are not interchangeable. A person may have the ability to submit queries without knowing the architecture. Another may know the architecture but not the parameters. A third may control a training component without ever seeing the deployed interface. The possible collusion paths depend on how those capabilities and knowledge sets overlap.
This is why “the attacker” is often too vague a subject for a serious threat model. The relevant entity may be a data contributor, a model supplier, a trainer, a service provider, or a client. The supplied framework represents an adversary through several characteristics: their role and objective, whether they are honest-but-curious or malicious, how they optimize, how they interact with the model, what they know about the model, and what they know about the data. The detail is useful only when it answers a concrete question: what resource can this participant provide that the other one lacks?
GenAI makes the handoffs even more visible. Foundation models may be pretrained on web-scale data, adjusted through supervised fine-tuning or human-feedback methods, released as weights or services, and then integrated into retrieval systems or agents. A downstream application may ingest documents at runtime, pass model output to a tool, and feed the tool’s result back into the next model prompt. NIST treats these as distinct attack surfaces, including supply-chain poisoning and indirect prompt injection. They are not empirical examples of every collusion chain in the SoK, but they demonstrate why an AI system’s effective boundary is larger than its model weights.
Two ways attacks can cooperate
The first pattern is train-test collusion. A train-time adversary uses poisoning or a backdoor to change the model that a later attacker will query. The later attacker may be pursuing evasion, membership inference, attribute inference, distribution inference, data reconstruction, or a discriminatory outcome. What passes between the stages is not necessarily a file or a message. It can be a change in memorization or in how differently the model behaves across records, datasets, or demographic subgroups. The later attack becomes more effective because the trained model now exposes a stronger signal.
The second pattern is test-time collusion. Both attacks occur against an already trained model, but the first produces something the second needs. Adversarial examples can serve as auxiliary observations for building another model. Membership inference or data reconstruction can identify records that make an auxiliary dataset more similar to the training set. Distribution inference can reveal a property of the training population. Model extraction can produce a surrogate with more similar behavior or architecture. The useful outcome is therefore portable knowledge, not only damage to the first target.
The distinction is important because the two patterns ask different defensive questions. For train-test collusion, a defender must ask how training manipulation changes later susceptibility. For test-time collusion, the defender must ask what the model reveals through one interaction and how that revelation can be reused. A query limit, for example, may matter not only because it makes one extraction attempt harder, but because it restricts the observations available to a different attacker. Conversely, direct model access or overlapping data may already satisfy a prerequisite, making a partner unnecessary.
The framework’s basic reasoning rule is an alignment test. Identify a factor that the first attack can increase or reveal, then check whether the second attack becomes more effective when that factor is present. Aligned effects suggest potential collusion. Opposing effects suggest that the first attack does not help the second, even if both involve the same model or stage. And if no common factor can be identified, the available evidence does not justify concluding that the attacks collude.
“Potential” is doing real work in that rule. A shared factor is not a guarantee of operational success. Query budgets, data quality, model architecture, timing, access assumptions, and the cost of coordinating can all break a chain. The SoK presents the framework as a heuristic based on correlations, not as a causal theory that predicts every attack outcome.
The question that changes the audit
The primary subject here is not an exotic new attack. It is the relationship among adversaries who encounter the same ML pipeline from different positions. That relationship is easy to overlook because security programs are commonly organized by stage or risk category: protect the data, test the model, limit the API, assess privacy. Those controls remain necessary, but each can be evaluated too narrowly if it ignores what its outputs make possible elsewhere.
A pipeline-wide threat model therefore begins with four questions. Who can change the data, code, model, resources, or queries? What does each participant learn from the system? Which outputs become reusable artifacts or changed conditions? Which later attack can use them, and under what assumptions? The answers do not prove that collusion will occur. They reveal where an isolated assessment has omitted an edge.
That is the central reason ML threat models break when adversaries collude: they often describe the nodes—poisoning, evasion, privacy, ownership—without describing the dependencies between them. The next step is to make those nodes and boundaries explicit enough to test. Once the pipeline’s state and authority are visible, the harder question becomes answerable: which participant can control, observe, or pass forward what the next participant needs?