The Information Economy of Test-Time Collusion

Test-time collusion is the deployment-side form of colluding adversaries in ML pipelines: one attacker learns or produces something that gives another attacker a better starting point against the same model. The focus here is the chain between an observed outcome and a later prerequisite. Understanding that chain lets a reader distinguish a real dependency from a mere sequence of attack labels, and recognize why the quality of an exchanged artifact can matter more than its quantity.

At deployment, an attacker may not be able to change the model’s weights or training data. They can still learn from the model’s responses. Those responses may reveal decision-boundary behavior, identify records that resemble training members, expose a property of the training population, or help produce a model that behaves like the target. A second attacker can reuse that result. The first attacker’s objective and the second attacker’s objective need not be the same; the shared value is the information or data generated along the way.

The four prerequisites are different kinds of closeness

The SoK describes four test-time factors that connect attacks. They are best understood as four different meanings of “close.”

The first is auxiliary-data quality and overlap. Many attacks train a shadow model or an attack classifier rather than operating directly from a single target response. The auxiliary dataset used for that training is more useful when it resembles, or partially overlaps with, the target’s training data. A party that identifies likely training records can therefore provide more useful material to a partner than a party that merely sends arbitrary queries.

The second is distribution similarity. Even when the auxiliary records are not the same records used to train the target, their distribution may need to resemble the target’s distribution. If the target was trained on one mixture of classes or demographic attributes and the shadow model is trained on a very different mixture, the shadow may learn the wrong signal. A distribution-inference attack can supply an estimate of the target’s population, allowing a later attacker to select or generate auxiliary data that is more representative.

The third is functional similarity of the shadow model. A shadow model is useful when it behaves like the target on the task that matters to the downstream attack. Functional similarity is about outputs and behavior, not necessarily identical internal construction. The fourth is architecture similarity, which concerns the structural form of the model. In some settings, knowing the target’s architecture helps produce a surrogate whose behavior transfers more reliably. In others, a functionally similar but differently structured model may be sufficient.

These factors overlap but are not interchangeable. A dataset can resemble the target’s distribution without containing any target records. A surrogate can match broad predictions while differing on the sensitive property a later attack is trying to infer. An architecture can be known while the parameters remain hidden. The SoK keeps the factors separate so that a proposed chain can state exactly what the first attack supplies and what the second attack still needs.