Defend the Edges, Not Just the Attacks

This series has examined colluding adversaries in ML pipelines as a problem of dependency: a training-time attacker can change a model condition, while a test-time attacker can produce data or knowledge that another attack needs. The final question is what a defender should do with that model when the evidence is heuristic, the pipeline includes third-party components, and no single control protects every property at once. The answer is to map and constrain the edges between roles, stages, artifacts, and objectives—not to abandon the familiar attack categories, but to stop treating them as independent endpoints.

The supplied SoK gives the central analytical move. An attack matters not only for its direct outcome but also for what that outcome makes possible. NIST’s taxonomy supplies the broader system context: AI security spans data, models, code, training, testing, deployment, runtime resources, and agents, with attackers varying in capability and knowledge. Together, the sources support a change in posture. A threat model should record what crosses each boundary, identify which later prerequisite it satisfies, test the relationship under explicit conditions, and then choose controls that reduce the value or reliability of that transfer.

The system to defend is a dependency graph

The graph begins with the lifecycle, but it does not stop at chronology. Data and code enter training. A trained model moves through evaluation and deployment. A service exposes observations to clients. In a generative system, a foundation model may be fine-tuned, aligned, connected to documents, and given tools. The system’s effective state includes not only the model parameters but also the datasets, prompts, retrieval resources, interfaces, permissions, and feedback loops that shape behavior.

The graph also includes people and organizations. A data provider, model supplier, trainer, service provider, and client may have different authority over the same system. Some roles may be combined in one organization; others may be separated across vendors. The role is relevant because it determines what an actor can plausibly change or observe. Training-data control can alter what the model learns. Model or source-code control can alter the learned system more directly. Query access can reveal behavior without granting permission to modify it. Resource control can affect what a retrieval system or agent consumes at runtime.

The second layer is the adversary’s knowledge. A black-box client may receive only labels or other limited outputs. A gray-box participant may know the architecture or data representation but not the parameters. A white-box participant may have much fuller access. The same downstream attack can therefore be direct in one threat model and collusion-dependent in another. A privacy attack that needs a shadow model may benefit from a model-extraction partner under black-box access, while a white-box attacker may already possess the relevant model knowledge.

The third layer is the artifact or condition that travels. In train-test collusion, the transfer may be a change in tail behavior, memorization, or distinguishability across data or subgroups. In test-time collusion, it may be auxiliary records, a distribution estimate, a surrogate model, or observations of the target’s decision behavior. Each transfer needs a receiving condition. If the second attack requires a particular kind of data overlap, model similarity, or distribution knowledge, the threat model should say so explicitly.

The fourth layer is direction and quality. The same upstream change can help one objective and hurt another. The SoK’s poisoning-to-model-extraction case is a useful reminder: in the tested setup, extending the tail reduced surrogate accuracy and fidelity rather than improving extraction. By contrast, extracted shadows improved distribution inference in the reported black-box comparisons. Reconstructed records improved several downstream privacy attacks in the tested settings, but membership inference declined at the highest replacement ratio. A graph that records only “poisoning,” “extraction,” or “reconstruction” loses the direction and shape that make the relationship meaningful.

A defensible edge therefore has at least five parts: a source actor and stage, a change or artifact, a receiving attack, an unmet prerequisite, and a condition under which the transfer improves the receiver’s outcome. It should also include timing and quality. An artifact may be too noisy, too narrow, too stale, or too expensive to produce. A partner may have a theoretical contribution that is redundant in the actual deployment because the receiving adversary already has direct access.

This makes the core judgment precise. Isolated attack analysis asks whether a node is vulnerable. Collusion analysis asks whether a path through the graph is viable and whether a partner adds unique value. Both questions matter.