From Lift Estimate to Certified Decision

This chapter turns the privacy-degraded report in the cross-channel advertising measurement system into a decision object. The focus is the difference between estimating a lift and certifying that it clears a business threshold. That distinction gives a team a disciplined answer when the report is compatible with more than one clean explanation: certify, reject relative to the threshold, or leave the claim unresolved.

A point estimate is attractive because it compresses uncertainty into one number. But when matching, linkability, attribution windows, suppression, and reporting noise have altered the evidence, that compression can hide the very question a decision-maker needs answered. The privacy-robust framework instead asks what clean experimental worlds are still possible given the report and the plausible signal-loss mechanisms.

The report defines a set of possible clean worlds

Let a released report for a segment contain the treated and control summaries that survived the reporting system. The clean incrementality value is the difference between the clean treated and control potential-outcome means. If the report were complete and the reporting mechanism known, the analyst could estimate that difference directly. Under privacy constraints, the analyst may know the released means without knowing how much clean outcome signal was omitted before those means were formed.

The framework represents that uncertainty with a compatibility fiber. It is the set of clean experimental worlds that could have produced the released report under at least one plausible signal-loss mechanism. A clean world includes the arm means and any hidden match, attribution, or linkability quantities needed to define the target. A mechanism includes the retention, suppression, and noise rules that connect that world to the report.

This way of thinking changes the order of analysis. Instead of selecting one presumed retention rate and calculating one corrected lift, the analyst first defines an ambiguity set of plausible mechanisms. The set may include a range of retained outcome signal for each arm and segment, an aggregation threshold, a reporting-noise scale, and restrictions justified by system documentation or diagnostics. The question is then uniform: which conclusions hold for every clean world and every mechanism still compatible with what was released?

Projecting the compatibility fiber onto incrementality produces a lower bound and an upper bound. The lower bound is the smallest clean effect that remains possible; the upper bound is the largest. In the simple bounded, nonnegative outcome case, if a released arm mean is the clean mean multiplied by a retention value between a lower and upper endpoint, the compatible clean arm mean runs from the released mean divided by the upper endpoint to the smaller of the outcome bound and the released mean divided by the lower endpoint. The treatment-effect interval comes from the smallest compatible treated mean minus the largest compatible control mean, and vice versa for the upper end.

The direction is intuitive but important. A high retention endpoint gives the smallest clean mean compatible with an observed mean, because little signal was lost.