The Pipeline Is a Network of Trust Boundaries
Colluding adversaries in ML pipelines become easier to reason about when the pipeline is treated as a system of trust boundaries rather than as a model with a single attacker. The question in this chapter is who can change, observe, or transfer state at each handoff—and how those conditions determine whether a second attack actually needs a partner. That map gives the reader a way to test a proposed collusion path instead of accepting it because two attack names appear next to each other.
An ML system may look like a finished model at the moment a user sends a query. Operationally, it is the endpoint of a longer chain. Data is collected and processed. Code and model components are supplied. A trainer turns inputs into learned parameters. Someone evaluates and owns the result. A service provider deploys it, and a client receives outputs. The boundaries between these activities determine what can be manipulated, what remains hidden, and what information becomes available later.
State changes hands before the model answers
Start with the artifacts rather than the organizational chart. Training data is not simply “inside” the model; it is selected, cleaned, labeled, and transformed before optimization begins. Source code determines how that optimization and evaluation are performed. A model provider may supply an architecture or a pretrained model. The trainer combines those elements and a configuration to produce a trained model. Evaluation data and metrics then influence whether that model is accepted for deployment.
The exact institutional arrangement can vary. One company may be the data provider, trainer, model owner, and service provider at once. Another system may obtain a foundation model from one organization, fine-tune it with a second organization’s data, host it through a third party, and expose it to thousands of clients. The roles in the supplied framework are therefore analytical positions, not a claim that every pipeline contains nine independent departments. They identify where authority and visibility enter the system.
At each handoff, ask what state crosses the boundary. A data handoff may carry records, labels, or a distribution that the receiver assumes is benign. A model handoff may carry weights and architecture, along with behavior that is difficult to inspect directly. A deployment handoff may expose predictions, confidence scores, or other model observations through an API. A client may not control training, but repeated interaction can still reveal properties of the trained system. Conversely, a model supplier may influence behavior without ever seeing the eventual client queries.
The dotted edges are the important ones. They represent more than the ordinary flow needed to operate the service. They represent control or knowledge that may cross a boundary and later become useful to a different participant. A threat model that records only the solid operational path can miss the security significance of those transfers.
Capability is not knowledge
The NIST taxonomy separates several capabilities that are easy to blur together. Training-data control means inserting or modifying records. Model control means modifying parameters or the model itself. Testing-data control means changing inputs at deployment. Source-code control means altering the algorithm or a dependency. Query access means submitting inputs and receiving predictions. These capabilities answer a question about what an adversary can do to the system.
Knowledge answers a different question: what does the adversary know about the system while doing it? A white-box adversary may know the architecture, parameters, training data, and configuration. A black-box adversary may know little beyond a public interface. Gray-box conditions lie between those extremes: an attacker might know the architecture but not the parameters, or know the feature representation and data distribution without seeing the training set. The same attack objective can therefore require different partners under different knowledge assumptions.
Interaction is a third dimension. A client with one opportunity to query a model faces a different problem from a client with a fixed budget of queries or a local, unlimited copy. The returned information also matters. A hard label reveals less than a vector of confidence scores; a model released with its weights exposes more than an API that returns only a final answer. Data knowledge adds another distinction: an adversary may have no overlap with the training set, partial overlap, or full overlap, and may or may not know the correct labels for evaluation data.
These dimensions should not be collapsed into a generic description such as “a sophisticated attacker.” Consider two parties interested in the same deployed model. The first has black-box query access and can collect informative responses. The second has an independent dataset but no architecture knowledge. Sharing the first party’s observations may help the second construct a closer surrogate. Now change the conditions: if the second party already has white-box access to the model, model extraction may add little; if they already possess the training data, an inference partner may be unnecessary. Capability can create a path, but existing knowledge can remove the need for one.
The distinction also prevents a common analytical error: assuming that an actor who can observe a system can change it. A client may infer a training-data property without controlling the training process. A data provider may insert records but have no ability to query the deployed model. Collusion becomes plausible when two parties can combine those asymmetric positions. It is not plausible merely because the pipeline contains both positions.
Roles explain where hidden coupling lives
The roles matter because they own different kinds of state. A data owner can contribute raw information that later becomes part of training or evaluation. A data provider can aggregate and process those contributions. A code provider can alter a library or training routine. A model provider can distribute an architecture or pretrained weights. A model trainer controls the operation that turns those inputs into learned behavior. The model owner controls the resulting asset, while the service provider controls how it is exposed. The client controls requests and can sometimes download a model rather than use an API.
Those descriptions are not accusations about any particular organization. They are a way to locate failure surfaces. If a trainer uses a component supplied by an external model provider, the model’s weights become a supply-chain dependency. If a service continuously updates from user interactions, the boundary between deployment and training may be porous. If an application retrieves documents from outside sources and puts them into a model’s context, the resource provider becomes part of the effective input surface even though they never call the model directly.
NIST makes this broader setting especially clear for generative AI. Foundation models can be pretrained on large, heterogeneous data collections, adjusted by supervised fine-tuning or human-feedback procedures, released as weights or hosted services, and integrated into retrieval-augmented generation systems or agents. Several organizations may perform those activities. The model is then not the whole application: a downstream developer adds prompts, retrieval, output handling, tools, memory, or planning. A weakness introduced at one layer can therefore become an assumption made by another.
The same logic applies to a runtime resource. In a retrieval system, a document is usually treated as information to be summarized or answered from. If the model does not reliably separate instructions from data, text inside that document can also influence the system’s behavior. NIST calls this an indirect prompt injection when a third party modifies a resource that the model later ingests. The example is not one of the SoK’s measured collusion chains, but it illustrates the same structural issue: a participant who controls an upstream resource can affect a downstream action without directly controlling the model.
Agents make the boundary more consequential because their outputs can select tools or construct inputs for the next step. An untrusted resource may not just distort a paragraph; it may influence a tool call, and the result of that call may be returned to the model as new context. The security analysis must then include the permissions of the tool and the state of the environment. The relevant question remains the same: what can cross the boundary, who can reuse it, and what action does it enable?
From an attack outcome to a later prerequisite
The word “outcome” can obscure this relationship if it is understood only as damage. In a collusion analysis, an outcome is any change or artifact that alters the next attacker’s starting conditions. A poisoned model may have different memorization or subgroup behavior. A query campaign may produce examples and responses. An inference attack may identify records or estimate a distribution. A model-extraction attack may produce a surrogate with more similar functionality. These outcomes can be valuable even when the first attacker’s original objective was different.
A prerequisite is the condition the second attack needs to work well. It might be auxiliary data that overlaps with the target’s training data, a dataset with a similar distribution, a shadow model that behaves like the target, or knowledge of the target’s architecture. The second attacker may obtain that prerequisite independently. If so, the first attacker’s contribution is redundant. If not, the first attacker may supply the missing piece.
This is the point at which a threat model becomes a dependency map. It should record the first attack, the artifact or changed condition it produces, the second attack’s required condition, and the assumptions connecting them. It should also record quality. A small amount of accurate auxiliary data may be more useful than a large amount of noisy data. A model that matches the target’s broad task but not its relevant behavior may be a poor shadow. A distribution estimate can be useful in one setting and too coarse in another.
The direction of the edge matters too. An attack can improve one downstream objective while making another harder. A poisoned model may expose more information to a membership attack but learn under-represented classes less reliably, which can reduce the quality of a model extracted from it. Treating “more vulnerable” as a single scale hides these differences. Collusion is about aligned changes in a particular factor, not about an overall impression that the system has become worse.
A threat model that can be tested
A useful map does not need to predict every attack. It needs to make its assumptions visible. For each participant, identify the role they occupy, their objective, and the capability they control. Then state their model knowledge, data knowledge, interaction budget, and whether they can adapt after seeing a response. Finally, name the artifact or model condition that is supposed to cross to the partner and the later prerequisite it satisfies.
That description supports two kinds of failure analysis. The first is a missing-edge failure: the proposed partner cannot actually provide the required data, knowledge, or model similarity. The second is a redundant-edge failure: the second adversary already has enough access to perform the later attack, so coordination adds little. Both are important. Overstating a collusion path can misdirect defensive resources; understating it can leave a chain untested.
The practical conclusion is that an ML pipeline is not merely a sequence of components. It is a network of authorities, observations, and learned states. Once those boundaries are explicit, poisoning can be analyzed for its effect on future susceptibility, and a query can be analyzed for what it reveals beyond the immediate response. The next question is how a train-time adversary changes the learned model’s conditions before a deployment-time partner arrives.