Does a Scenario Ever ‘Win’?

The Federal Reserve's historical evaluation asks a deliberately awkward question: when later data arrive, was an alternative scenario closer to what happened than the baseline? This chapter focuses on what that comparison can teach about directional plausibility and what it cannot prove about forecasting skill. The reader's task is to interpret the 16% result, the turning-point cases, and the large-error periods without confusing an ex-post winner with a path that was knowable in advance.

What the historical test actually measures

Alternative scenarios are not ordinarily predictions. They are conditional illustrations prepared to help policymakers think through risks. That makes performance evaluation difficult: if a scenario says what could happen if a particular risk materializes, what should count as success when the risk does not occur, occurs in a different form, or arrives at a different scale?

The Federal Reserve paper approaches the question retrospectively. It examines the period from 2000 through the onset of the pandemic, when alternative simulations had become more numerous, and compares the scenario paths with realized macroeconomic outcomes. The comparison uses real GDP growth and core inflation, expressed as deviations from the baseline forecast. At a one-year horizon, the authors identify which scenario—including the baseline—came closest to the subsequent outcome for each meeting.

“Closest” has a specific meaning here. The authors use a distance metric based on equally weighted squared deviations in GDP growth and core inflation. If the scenario misses GDP or inflation, the error is squared; the two errors are then combined with equal weight. A path that is moderately wrong on both dimensions can therefore be judged differently from one that is close on GDP but far off on inflation. The metric makes the comparison reproducible, but it is still a choice about what matters and how much each variable matters.

That choice creates a useful but narrow lens. It treats one percentage-point miss in GDP growth and one percentage-point miss in core inflation as comparable after the variables are put into the study's common measure. It also emphasizes larger misses because they are squared. A policymaker might care about other dimensions—financial stress, unemployment persistence, timing within the year, or the costs of an inflation overshoot—and those concerns are not captured by this particular ranking. The evaluation is strongest when it is read as a transparent test of one set of macroeconomic coordinates, not as a complete score for a scenario.

The number of scenarios at a meeting matters as well. If six or seven alternatives surround a baseline, one path will be closest by construction even when every path is far from the outcome. The comparison becomes more informative when the realized point falls inside a range that resembles the scenario spread and when the scenario's narrative matches the development that occurred. The numerical winner is therefore a starting point for asking whether the set contained a plausible branch, not a final verdict on the staff's judgment.

The realized data also carry an important qualification. The comparison uses current-vintage values, incorporating later data revisions, to represent the best subsequent understanding of what happened. That is appropriate for a retrospective exercise, but it gives the evaluation information that policymakers did not have at the original meeting. The test asks which published path later looked closest, not what a policymaker could have selected using only information available at the time.

The authors explicitly describe the result as a heuristic measure of ex-post plausibility rather than a statistical horse race among competing forecasts. That wording should govern how the headline numbers are read.