Choosing Better Evidence: Training Points, Cross-Validation, and Error Bars

Once a cross-screen system can bound an unobserved reach, the next decision is where to spend limited measurement effort and how much to trust a model-based refinement. This chapter compares those choices through the subset-reach framework: measure the subset whose uncertainty matters, tune a point-estimation model against held-out observations, and report an error signal without turning a synthetic experiment into a production guarantee. The practical goal is information gain, not a larger pile of platform reports.

Measure the uncertainty that changes the decision

An additional subset observation is valuable when it constrains the overlap that matters to the decision. The subset-reach framework uses the model-free bounds for this purpose. For each candidate subset that has not yet been measured, it calculates the gap between the largest and smallest feasible reach. The candidate with the widest gap is selected as the next training point, measured, and added to the observed set. The process can then be repeated.

This is a more disciplined use of measurement budget than adding another platform because it is available. A buyer deciding whether streaming adds people beyond a television-and-mobile plan does not primarily need another total reach figure. They need evidence about the subset or set difference that controls that incremental question. The bound gap is a way to identify where the current observations leave the most room for competing answers.

The rule is a heuristic, not a universal optimum. The widest gap for one candidate may not be the observation that most narrows a particular incremental-reach curve. A subset can have high uncertainty but share little structure with the target the buyer actually cares about. Another subset can have a smaller headline gap while constraining the target’s critical primitive regions. The paper’s adaptive experiment explicitly notes that selecting the largest uncertainty is not necessarily the same as minimizing the area of a chosen curve.

That limitation is not a defect in the idea of adaptive measurement. It identifies the criterion that must be made decision-specific. If the decision is total campaign reach, select observations that constrain the full union. If the decision is the incremental value of one platform after a specified plan, select observations that constrain that platform outside the plan’s union. If the decision is a sequence of placements, the target may be a family of set differences rather than one subset. “More information” becomes meaningful only after the target is named.

This prevents a common procurement mistake: commissioning a broad overlap study and assuming its largest aggregate statistic will answer every downstream question. A study can be excellent for total union reach and weak for platform-level increment, or useful for one reporting period and poorly matched to another. The measurement brief should name the target subset, reporting window, universe, and decision threshold before the candidate observations are ranked.