How Strong Is the Intervention Evidence?

The subject here is the causal strength of the virtual-contact findings, not whether the program’s results are interesting. This chapter examines the comparison group, the people who remained in the analysis, and the difference between the primary repeated-measures results and a supplementary difference-in-differences analysis. The reader’s task is to rank the evidence without collapsing it into a binary choice between “controlled” and “uncontrolled.”

A comparison group is not a randomizer

Study 1 showed that participants reported warmer attitudes and more self–outgroup overlap after the program, but it had no control condition. Study 2 improved the design by adding students who did not participate in the current program and by comparing change over time between the two conditions. That is a meaningful advance. A control group gives us a picture of what happened during the same broad period without the focal intervention, making it possible to ask whether the program group changed more than a comparison group.

But the strength of that question depends on how the groups were formed. Study 2 was a field quasi-experiment, not a randomized trial. The intervention students came from classes taught by professors who had previously enrolled students in Soliya. Control students came from classes taught by those professors but were not currently participating. Class sections determined the condition, while intervention students were mixed across classes into small discussion groups. The arrangement makes the program accessible in an ordinary educational setting, but it does not ensure that students in the two conditions were equivalent before the program.

The imbalance is visible before looking at the outcomes. Of the 2,382 enrolled students, 1,259 were American students, with 1,087 in the virtual-contact condition and 172 in control. The other 1,123 students came from the MENA and Southeast Asian regions, with 956 in virtual contact and 167 in control. After completing both surveys and passing attention checks, the analytic sample retained 855 American and 770 Muslim participants in the intervention condition, compared with 94 American and 111 Muslim participants in control. A comparison with a much larger treatment group and a much smaller control group can still be informative, but it has less protection against accidental imbalance and makes the control group more sensitive to recruitment and retention differences.

The control participants also had a different experience of participation. They completed surveys during corresponding time periods on their own time, rather than during scheduled program sessions, and they did not receive the intervention or the same compensation. That difference is not a minor administrative detail. It can affect motivation, attention, expectations, and the likelihood of completing the post-program survey. The result of interest is supposed to be the effect of virtual contact, but the observed contrast may include differences in institutional engagement, researcher contact, convenience, and perceived benefit.

This is why a comparison group should be understood as a constructed counterfactual, not a magical substitute for random assignment. It improves the question from “did participants change?” to “did participants change more than these other students?” It does not automatically answer whether the groups would have changed in parallel without the program. That assumption is especially important in a design with class-based condition assignment, unequal group sizes, and no preregistration.

Retention makes the counterfactual more complicated. In Study 2, attrition was predicted by participants’ ethno-religious background and condition, and by pre-program dehumanization.