How the Experiment Separated Polish from Expertise
This chapter examines how the ifo/CESifo study tested sender-side ChatGPT assistance in hiring and entrepreneurship. The important design question is whether researchers can observe the difference between a pitch that looks better and a receiver who becomes better at identifying expertise. Understanding the treatment, comparison, and outcome measures makes the later claims about screening error interpretable rather than merely impressive.
A controlled comparison, not a real hiring funnel
The study was designed as a controlled screening experiment. One group of participants acted as senders: they wrote pitches intended to persuade an employer or investor. Another group acted as receivers: they judged the pitches and inferred whether the sender had relevant expertise. Because the researchers measured the senders' prior experience, they could compare a receiver's belief with a known experimental classification.
This arrangement creates a counterfactual that ordinary applications rarely provide. In a real hiring process, an employer sees one candidate's application and does not know what the same person would have written under another production condition. In the experiment, each sender developed multiple pitches. The design included pitches produced without ChatGPT and pitches produced after the sender was allowed to use it. The comparison is therefore within a structured task rather than a loose contrast between different populations of writers.
The receiver was not asked only whether a pitch sounded persuasive. They evaluated the pitch, estimated the probability that its author was an expert in the relevant domain, and rated how much they would pay for a more costly form of evidence. Receivers were recruited with relevant experience and evaluated randomly assigned pitches. They were also told whether ChatGPT assistance was involved. That disclosure matters: the experiment isolates the effect of a sender's AI-assisted signal in a setting where the receiver knows the condition, rather than silently mixing uncertainty about whether AI was used into the main treatment.
The resulting data contain 343 senders, 801 receivers, and 6,440 sender–receiver evaluations. Those counts describe the study's analytical sample, not a national labor market or a representative venture-capital population. Their value is different: they create repeated observations in which a sender's expertise, pitch condition, receiver judgment, and outcome measure can be linked.
The experiment does not make the receiver omniscient. It gives the researcher an outcome against which the receiver's inference can be judged. That is the central advantage of the design: “the pitch was rated highly” and “the sender was actually an expert” remain separate observations.
Two settings, one screening logic
The hiring setting used two expertise domains: data science and management consulting. A sender had prior work experience in one of the two domains, then wrote a pitch for a role in both. A receiver with hiring experience in the relevant industry evaluated the pitch and judged whether its author had previously worked in the target role.
The entrepreneurship setting followed a parallel structure with different content. Senders had prior entrepreneurial experience in one of two sectors, retail or education. They wrote a pitch for a new business idea in the sector where they had expertise and another in the sector where they did not. Investors with experience in those sectors evaluated the pitches, judged whether the author had experience in the relevant industry, and rated the pitch's quality.
These settings test the same informational relationship without pretending that employment and investment are identical. In hiring, the receiver is screening for a candidate's experience in a job domain. In entrepreneurship, the receiver is screening for an entrepreneur's familiarity with an industry and the plausibility of a business idea. Both require inference from a written pitch, but the meaning of a good outcome differs. A candidate's prior role experience is not the same variable as a venture's future survival.
The parallel structure is valuable for a more limited reason. If sender-side ChatGPT changes receiver accuracy in both settings, the result is less dependent on one particular kind of pitch. It still remains a result of these controlled domains, not proof that every form of hiring or venture finance will respond identically. The researchers use the two contexts to test robustness across settings; they do not erase their substantive differences.
The treatment comparison also has a precise interpretation. It is not “human writing versus writing created entirely by a machine.” Senders wrote pitches and then received the option to use ChatGPT to rewrite or improve them. The relevant condition is the availability and use of assistance within the experiment. This matters because real-world AI use can range from light editing to extensive iteration, while the experiment is designed to compare a bounded treatment rather than catalog every possible workflow.
What was actually measured
The first outcome is the receiver's belief about expertise. Receivers selected a probability interval representing how likely the sender was to be an expert. The study used five buckets, from very low to very high probability, and converted those intervals to midpoints for analysis. This measure captures what the receiver believed, not whether the belief was correct.
The second set of outcomes measures error. One measure uses the variance of a probability belief: a receiver who chooses a middle probability is more uncertain than one who chooses an extreme probability. Another compares the receiver's probability with the sender's actual expertise and squares the difference. The first reflects uncertainty in the belief; the second reflects distance from the observed truth. Both are attempts to measure screening accuracy, but they are not the same statistic and should not be treated as interchangeable.
The third outcome is the level of the signal: the receiver's 1–5 rating of pitch quality. This is the measure most likely to resemble an ordinary reader's reaction. It tells us whether the pitch appears stronger, more polished, or more sophisticated. It does not by itself say whether the sender's underlying expertise was identified correctly.
The study also uses a natural-language measure based on the sophistication of the writing. The researchers transform a readability score so that higher values represent more sophisticated language. This provides a second way to observe how the text changes, but it remains a property of the language, not a direct measure of domain competence. A text can become more difficult or formally polished without supplying better evidence about the person who produced it.
The fourth outcome is dispersion. Instead of looking only at the average pitch rating, the researchers measure how far individual ratings or language scores sit from the mean within each condition. This captures whether pitches become more varied or more alike. Dispersion matters because variation may help a receiver distinguish experts from non-experts. A higher average with a narrower spread could mean that the messages have become more uniformly impressive while losing some of their discriminating content.
Finally, receivers stated how much they would pay for an additional signal. In hiring, the example was a background verification. In entrepreneurship, it was market research conducted by potential customers. This is a behavioral proxy for trust in the pitch: if receivers want more expensive evidence after seeing an AI-assisted message, they may regard the text as less sufficient. It is not proof that they would actually purchase the service in a real organization, and it does not establish that the alternative is automatically more accurate or fair.
The measures are deliberately plural because the theory makes plural predictions. AI could raise pitch ratings, change receiver uncertainty, alter error relative to actual expertise, compress the distribution, and increase demand for other evidence. No single score can answer all of those questions.
Why the comparison needs an observed truth
A receiver's belief and a sender's actual expertise play different roles in the design. Suppose receivers assign an average 45 percent probability of expertise to a group of senders. That number alone cannot tell us whether the receivers performed well. If exactly half of the senders are experts and the receivers identify the right people, the average could be close to 50 percent. If receivers assign similar probabilities to experts and non-experts, the same overall average could hide substantial misclassification.
The study can examine this because sender expertise was built into the task. Each sender wrote in a domain where they had relevant experience and another where they did not, and the analysis used those differences to compare predicted expertise with actual expertise. The key question is not whether receivers liked the pitch. It is whether their judgments tracked the sender's domain knowledge.
This distinction also explains why a receiver's quality rating remains useful without being decisive. A pitch rating tells us what the receiver saw in the message. The expertise comparison tells us whether that impression was diagnostic. If the rating rises for both experts and non-experts, the output may be more attractive but less selective. If it rises primarily for experts, it may both look better and preserve more information. The experiment is designed to distinguish these patterns.
The design does not eliminate every source of uncertainty. Receivers may disagree about what expertise looks like. A binary classification simplifies a continuum of knowledge. A sender's performance in a short writing task may not represent their performance in a real job or venture. But these limitations are compatible with the central test. The study does not need to measure all of human capital to ask whether, under its defined task, AI assistance changed the connection between known expertise and receiver judgment.
What the design can and cannot establish
The controlled setting supports a causal comparison for the treatment as implemented. It allows the researchers to ask what changed when senders could use ChatGPT while holding the task structure and repeated sender–receiver relationships in view. It also supports comparison across hiring and entrepreneurship rather than relying on one anecdotal pitch.
The scope remains important. Participants came from Prolific, and the study was an artificial exercise rather than a high-stakes hiring round or a live investment decision. Expertise was treated as binary. Everyone in the treatment condition had access to the technology, so the experiment does not reproduce a labor market in which some candidates have access and others do not. The study examines one-shot screening rather than the later performance of a hired worker or funded company.
These limits do not invalidate the findings; they identify what kind of evidence they are. A clean experiment can reveal an initial causal mechanism without establishing the size of an effect in every organization. Field evidence could change the interpretation if long-run performance, unequal adoption, richer measures of expertise, or different kinds of signals produce a different relationship.
The design also leaves open whether organizations will continue screening for the same conception of expertise once AI is widespread. Employers might decide that effective use of AI is itself part of the job. Investors might care less about unaided pitch production and more about how founders validate assumptions with customers. Those are plausible changes in what is being screened, but they are not outcomes measured by this experiment.
The useful conclusion at this stage is methodological. To know whether AI strengthened a signal, we need more than a better-looking pitch. We need a measure of the underlying capability, a comparable production condition, and outcomes that separate perceived quality from accuracy, variation, and demand for corroboration. With those distinctions in place, the study's central result can be read as a pattern across measures rather than as a single verdict about ChatGPT.