Overview

The final rule is conditional: define the capability or outcome a hiring or investment decision needs to learn, identify where AI changes production or evaluation, and test whether the output still tracks that target. If separation is strong, use the pitch as one input and monitor it; if it is weak or unclear, add independent, capability-relevant evidence. Review subgroup effects, outside-information exposure, added costs, and later outcomes without treating the studies as a universal policy.

Learning outcomes

Start with the latent capability or outcome, not a convenient polish, language, fundraising, or prediction metric.
Audit whether AI changes the sender, evaluator, available information, or several links at once.
When separation is weak or unknown, add independent evidence that tests the target and monitor subgroup effects and costs.

Keywords

Generative AIValidationScreeningEvidenceHiringInvesting

What to Do When the Signal Changes

This final chapter answers the series' central question: when generative AI improves a hiring or startup-investing pitch, when has it improved the evidence and when has it only improved the appearance? The answer is a conditional decision rule, not a verdict for or against AI. A manager, investor, or system designer must identify the capability or outcome at stake, test whether the AI-shaped output still distinguishes it, and add proportionate evidence when the relationship is uncertain.

The defensible answer to the title

AI improves a signal when the assistance makes the relevant capability more visible than it was before. It weakens a signal when it makes outputs more similar across people with different levels of that capability, or when an evaluator mistakes a polished surface for evidence it does not contain. The same tool can do either because the decisive relationship is not the average quality boost. It is the relationship between expertise and the boost received.

That is the most defensible answer to the tension in the title. The ifo model predicts that AI can improve screening when experts benefit substantially more than non-experts, and can reduce accuracy when the gains compress the difference between them. Its pooled experiment found the latter pattern in its controlled hiring and entrepreneurship settings: pitch ratings rose, language sophistication rose more strongly, signal dispersion fell, screening errors increased by roughly 4–9%, and receivers expressed greater willingness to pay for additional evidence. The result is a coherent warning, but it is not a universal failure rate.

The subgroup analysis prevents the warning from becoming deterministic. The estimated expertise–AI-boost relationship was negative for the full sample and more negative for the English-speaking-country group, but positive for senders from non-English-speaking countries. The study found some evidence of improved screening accuracy in the latter context, while other outcomes did not all reverse significantly. The relevant lesson is that an average treatment effect can conceal a mechanism that changes by population.

The evaluator-side venture study adds a second qualification. Selected LLMs reportedly mirrored real fundraising outcomes strongly, but apparent survival-prediction accuracy was largely associated with prior exposure to some ventures' digital footprints rather than genuine reasoning under uncertainty. Scientific, contextual, and social cues could improve evaluations while also activating anchoring and herding. A result can therefore be accurate for a reason that does not generalize to a new case.

Taken together, the sources distinguish four properties that are often collapsed into one. Output polish describes how impressive or sophisticated a pitch looks. Signal separation describes whether the output distinguishes experts from non-experts. Outcome matching describes agreement with a recorded event such as fundraising. Reasoning under uncertainty asks whether the evaluator used evidence that would support a sound judgment when the relevant future was not already known. A system may perform well on one property and poorly on another.

Start with the trait, not the metric

Before choosing an AI-related score, define what the decision is trying to learn. This sounds obvious, but the signal problem often begins when a convenient metric quietly replaces the original objective.

In hiring, the target might be technical expertise, the ability to diagnose a customer problem, or the capacity to perform a role with AI available. Those are not identical. A polished cover letter may be relevant to written communication, but it is not automatically a measure of technical competence. A language-sophistication score may reveal that the text has changed; it cannot by itself establish that the candidate can perform the work.

In startup investing, the target might be customer understanding, the plausibility of a market hypothesis, execution capability, or the venture's survival prospects. A fundraising outcome is informative about what happened in the capital market, but it is not a pure measure of venture quality.

Published by DeepMox

FAQ

Why should a screening process define the target capability before choosing an AI metric?

It should define the target first because polish or language scores may measure presentation rather than the technical, customer, or execution capability the decision needs. A metric is useful only when it is tied to that objective.

Should a hiring test allow AI when the job uses it?

It may be more decision-relevant to test the candidate with the relevant tool when AI use is part of the job, rather than preserve an unaided writing ritual. The sources present this as a design implication, not a tested policy.

How can investors audit an LLM’s venture evaluation?

They can check what public information the model could have encountered, which cues were supplied, and whether the output tracks the intended investment objective. Human agreement is a benchmark, not proof of validity.

Previous7 minGenerative AI as a Venture Evaluator

All chaptersBack to Topic

Generative AI Screening: How to Validate Signals

What to Do When the Signal Changes

The defensible answer to the title

Start with the trait, not the metric

FAQ