How a Model Can Pass the Audit and Fail in Practice

This chapter diagnoses the data-forging vulnerability in cryptographic model certification: how a provider can produce a model that looks accurate or fair on known audit evidence while behaving differently on fresh data. The useful judgment is to separate a broken cryptographic protocol from a protocol that faithfully verifies a statement that was too narrow. The case study’s accuracy, fairness, privacy, and statistical-detection examples show the causal path, while its experimental limits determine how broadly the conclusion can be applied.

The cause is not a broken proof

The surface symptom is a reassuring certificate. The auditor receives a proof that a committed model met an empirical condition on the audit dataset. The model may be bound to the commitment, and the provider may have followed the specified training computation honestly. Yet the model can still behave badly away from the points used for the audit. The apparent contradiction disappears once the source of the failure is located: the provider was allowed to shape the model or its training data after learning what the test would contain.

This adversary is different from the external attacker usually described in a data-poisoning problem. In ordinary poisoning, an outside attacker tries to corrupt some of the data used by a provider who is attempting to learn an accurate model. The provider is the defender, and the attacker’s poisoned contribution is often constrained relative to clean data. In the cryptographic model-certification setting, the provider is the malicious party. The provider can choose the model parameters and training data, may corrupt all of the training data, and wants the resulting model to pass the audit while pursuing a conflicting objective in deployment. The auditor is the defender, but cannot simply inspect the proprietary model or data.

That reversal changes the security question. A commitment can stop the provider from replacing the model after certification. A zero-knowledge proof can stop the provider from claiming a computation that the proof relation does not support. Neither prevents the provider from selecting a model-building process that is already adapted to a known audit set. If the evidence is public or disclosed before commitment, the provider can treat the audit as a design constraint rather than as an independent test.

The paper formalizes this situation as an attack game. The audit dataset is sampled, the provider sees it, and the provider then outputs a hypothesis and training dataset. The provider wins if the model passes the empirical predicate on the audit data, fails the intended distributional property on fresh data, and satisfies a utility condition representing the provider’s goal. That final condition matters because the issue is not random overfitting alone. A provider may want a model to deny more claims, favor one outcome, or otherwise optimize an interest that conflicts with the auditor’s stated property while preserving an acceptable local score.

The central mechanism is the creation of two behavioral regimes. Around the known audit points, the engineered training data encourages the model to produce the responses the auditor expects.