From Prediction to Evidence

In the Egyptian study of formal credit and SME performance, the difficult question is not whether a larger enterprise loan has an average effect. It is whether information available before lending can identify firms whose responses to that loan differ. This chapter explains the study’s sample-split machine-learning design, its two main ways of testing heterogeneity, and the correction that changed several reported figures. The goal is to understand what a predictive ranking establishes—and what it still cannot tell an allocator about an individual firm.

Why an individual treatment effect must be predicted

For any one business, there are two relevant outcomes that cannot both be observed at the same time. One is the firm’s performance if it receives the larger loan; the other is its performance under the smaller comparison condition. The experiment can observe one of those paths for that firm and use randomized comparisons to estimate average and group-level differences. It cannot simply subtract the unobserved alternative from the observed outcome and call the result that firm’s causal effect.

The researchers’ solution begins with information collected before the loan-size treatment. Baseline business characteristics, cognitive measures, risk responses, financial-literacy measures, and psychometric questions are used to predict how a participant might perform under each condition. The difference between those two predicted outcomes is a predicted individual treatment effect. It is a forecast of relative response, not a directly observed fact about the firm.

That distinction matters because a model can be made to look persuasive by learning from the same outcomes used to judge it. If the model sees a firm’s post-loan profit while it is being built, it can fit quirks of that sample rather than discover a ranking that works for new observations. A ranking that survives a genuinely separate test is more informative: the relevant question is whether baseline information predicts a treatment-response difference that appears again in outcomes the model did not use to train itself.

The appendix describes a training-and-testing design for that reason. A random half of the data is used as the training sample. The researchers fit predictive models there, using baseline data to estimate treatment and control outcomes and then form predicted treatment effects. The resulting prediction is carried to the other half, the testing sample. In that testing sample, the predicted treatment effect is interacted with the actual treatment indicator and compared with observed business outcomes. The model is not allowed to evaluate itself on the observations that created its ranking.

This is a causal test with a predictive input, not a claim that the model has recovered a hidden personal attribute. Randomization supplies the treatment contrast. Baseline prediction supplies a proposed way to sort firms by expected response. The testing regression asks whether firms predicted to respond more strongly actually show a different treatment response. Without the randomized treatment, the prediction could be confounded with ordinary differences between profitable and less profitable firms. Without the holdout test, the prediction could be an in-sample story.

The procedure repeats this exercise rather than relying on one arbitrary split. The appendix says the analysis is rerun 100 times with different training and testing partitions. It also uses four common machine-learning approaches and applies adjustments for the repeated splits and the multiple methods. The point is not that one algorithm is intrinsically economic or that more algorithms guarantee truth.