From Hypothesis to Pilot and Main Experiment
AutoResearch treats an idea as ready to run only when it can be turned into a falsifiable, resource-bounded experiment. This chapter follows that transition: how a planner turns a proposed mechanism into measurable criteria, how a pilot decides whether the idea deserves more investment, and why the main experiment is a new evidentiary step rather than a larger copy of the first run. The reader’s practical task is to judge whether a plan can produce an interpretable answer before scale makes ambiguity expensive.
A plan is a contract for finding out
A model-generated idea usually arrives as a story about what might work. A plan must convert that story into obligations. AutoResearch’s planner is required to state the hypothesis, define success criteria, divide implementation into bounded modules, identify risks, and specify what observations would show that the idea is not valid. The plan is not meant to describe every line of code. It is meant to make the research question concrete enough that another role can implement and review it without silently choosing a different question.
The contract begins with a primary metric. It must identify a measurement, a binary threshold, a dataset, and the reason that the measurement bears on the hypothesis. “Improve performance” is not enough because it leaves the evaluator free to choose the task, the comparison, and the amount of improvement after seeing the result. A statement such as “the proposed method reaches at least a specified value on a specified evaluation set” can be checked as met or not met. The exact metric and threshold depend on the idea; the important property is that the decision rule is fixed before the result is interpreted.
The planner must also include an auxiliary criterion that guards against a misleading success. The project gives examples such as a training-time ceiling or a minimum sample requirement. This matters because a primary metric can be satisfied by changing an uncontrolled variable. A method may appear better because it used far more computation, saw a different amount of data, or was evaluated on too few cases. An auxiliary condition does not solve every confounder, but it forces the plan to state which costs or anti-cheating checks are part of the claim.
Modules add a different kind of boundary. The planner divides the work into a small number of independent pieces, each with a task, a file scope, and dependencies. That decomposition tells the coder what must exist and tells later review what the implementation is supposed to cover. It also limits an all-purpose agent response: if a plan requires data preparation, a baseline, and a core intervention, those responsibilities should not disappear into one vague instruction to “implement the method.”
The plan’s budget is part of the claim as well. The draft contract uses conservative defaults for the number of runs, revisions, and GPU hours, while retaining a policy for what happens after the pilot. A plan that can succeed only by consuming unlimited retries is not a falsifiable plan; it is an open-ended search. A bounded budget does not make an experiment scientifically sufficient, but it makes the cost of learning visible and gives a researcher a reason to stop or revise rather than drift into accidental scope expansion.
Before code is treated as ready, the plan passes a review gate. That gate asks whether the hypothesis is clear, the criteria are measurable, the modules are executable, and the idea’s hard constraints are respected.