The Experiment Tree: How OpenResearch Preserves Comparability
OpenResearch uses an experiment tree to answer a deceptively simple question: when a result improves, what exactly did it build on? This chapter focuses on the mechanism that connects a project baseline to parent and child hypotheses, and on the rules that keep a branch's result interpretable. Readers should come away able to tell when two variants are genuine siblings, when a new idea should descend from a winner, and why a failed execution is repaired rather than mistaken for a scientific answer.
The tree is not a decorative project-management view. It is a record of comparison. A branch says which source state a hypothesis started from; a parent says what earlier decision the child claims to preserve; a run says whether the branch actually answered the question. Without those relationships, a collection of Git branches can preserve history while still making progress impossible to interpret.
A child is a claim about what the parent established
Each OpenResearch experiment is a local record with a project, an optional parent, a branch, a title or description, and a run command. A root experiment branches from the project's baseline branch. A child branches from the parent's tip. The code relationship is straightforward, but its meaning is stronger than “copy the current files”: the child is supposed to change one decision while retaining what the parent established.
That is why the title and description matter. They name the question the node owes, not merely the branch's location. “Try a different optimizer” is a hypothesis-shaped description; “miscellaneous changes” is not. The first tells a later reader what was intended to vary and what should remain comparable. The second leaves the parent-child relationship without a clear interpretation, even if the Git history is technically valid.
The parent also carries an execution convention. A child inherits its parent's run command unless a deliberate project or experiment setting supplies one. The intended workflow then holds the command and environment as a fixed contract while the branch changes code or configuration. This narrows the comparison: the output is meant to reflect the branch's research change rather than an unannounced change in how the code was invoked.
That rule does not mean every experiment must use the same command forever. It means that a change to the evaluation recipe is itself meaningful and should not be hidden in a launcher's ad hoc arguments. If the command changes between variants, the researcher has changed more than the code under test. The tree can still record that decision, but the resulting comparison must be described as a different question rather than as a clean test of one code change.
Branch depth therefore has a specific interpretation. Imagine that the baseline is a model's starting configuration. A first round considers several alternatives to one decision, such as different optimization schedules. If one alternative is selected, a later round might test an architectural change while preserving the selected schedule. The architecture branch should be a child of the schedule winner, not another child of the baseline. Its depth says that the later question is conditional on the earlier decision.
This is the useful kind of lineage: not every descendant is automatically meaningful, but a meaningful descendant makes an explicit claim about what it keeps. The repository can tell us that one branch forked from another; the researcher must still state what that fork means. OpenResearch's tree rules are designed to make that statement unavoidable.
The distinction is visible in how a node is created. On an empty project, a parentless experiment establishes the first root. Once a root exists, an ordinary child creation can default to that project root, while an explicit parent selects a different point in the tree. The code then creates a new experiment branch from the chosen fork point and stores the relationship alongside its title, description, and command. This makes parent selection a research decision rather than a side effect of whichever checkout an agent happened to have open.
The project base branch is not the same thing as an experiment branch. It can remain mutable for the publication surface, while the orx/* branches hold the code associated with experiment nodes. That separation matters when an agent needs to update documentation or prepare a report after a result has been recorded. Such work should not quietly rewrite the branch whose commit defined the measurement. A child gives the researcher a place to combine later changes without changing the historical node.
The tree also makes restraint observable. A description that says “change only the schedule” creates a narrower obligation than a description that bundles an architecture rewrite, data change, and new evaluation metric. OpenResearch cannot inspect the quality of the hypothesis from the branch relationship alone, but the title, description, parent, and fixed command give a later reader enough structure to ask whether the claimed comparison matches the actual change. Lineage is useful precisely because it turns an informal intention into something that can be checked against the source and run.
From unresolved run to frozen lineage
The tree distinguishes a node that has not yet answered its question from one that has. Before a run establishes a baseline or tests a hypothesis, the node is provisional. A run that dies because of a missing dependency, an implementation error, a timeout, or an unintended hardware problem has answered neither the scientific question nor the node's intended measurement. The right action is to repair the same branch and retry, within the workflow's bounded repair policy.
This distinction prevents infrastructure failure from polluting the tree. If a broken launch immediately created a new child, the history would imply that the child represented a new idea when it really represented an attempt to make the old idea executable. Repairing the provisional node keeps the scientific question stable while the implementation problem is corrected. It also preserves a truthful account of how many real hypotheses have been tested.
Once a run does answer the node—whether the result is good, bad, or otherwise informative—the node becomes frozen. Its branch and history are no longer a workspace for later edits. A disappointing result is still a result. Editing the branch after the fact would change the source that produced the answer and make the historical record ambiguous. To pursue a new idea, the researcher creates a child and makes the new change there.
Freezing is therefore not a judgment that the branch is correct. It is a judgment that the branch has become evidence about the question it represented. The rule is especially important when a result is negative. A failed hypothesis may tell the researcher not to spend another round in that direction; rewriting it until the number improves would remove exactly the information that should guide the next decision.
The tree's shape follows from these meanings. Options that compete within one decision should be siblings because they are co-equal alternatives. New decisions should branch from the confirmed winner because they build on a resolved choice. OpenResearch describes the resulting form as stacked bushes: each round has a small fan of options, and the next round descends from the option that the evidence supports. A flat fan keeps every result measured against the start, so gains do not accumulate. A long single-child chain manufactures depth without showing that each step builds on the one above it.
The diagram shows why “winner” is a structural role rather than a permanent label of quality. The result is confirmed only relative to the question and evidence of its round. The winner becomes the next parent because the next hypotheses preserve its tested change. If later evidence overturns the choice, that new evidence can be represented by another child or a later round; the earlier node still records what was learned at the time.
This structure also gives waiting and resource allocation a research meaning. The intended autoresearch loop does not require launching every possible variant before reading any result. The researcher launches a round, returns when a run completes, rereads the project's run states, and inspects the log and branch difference. A clear answer may justify promotion. A mediocre but valid answer may justify refilling the round. A failure that answered nothing calls for repair. A goal reached or an exhausted branch justifies stopping. The tree and the run loop are connected: the state of one determines the meaning of the next action.
The first completion is a signal to analyze, not a license to ignore the rest of the round. A run can finish while another run has become terminal during the analysis window. The workflow therefore treats the wait operation as a wake-up signal and rereads the project's run records as the source of truth. That small operational rule protects the tree from a subtle accounting error: acting on the first visible completion while leaving another answered node unexamined or launching a new child from incomplete information.
The same discipline applies to comparing branches. A status line can tell the researcher that a run is done, but the evidence workflow requires reading the log and checking that it identifies the effective variant, final metric, compact summary, and relevant trajectory. A branch diff then answers what changed relative to the parent. Only after those two views agree can a result support a promotion or a refill. The tree supplies the question; the log and diff supply the material for answering it.
This is also why a small round is not merely a productivity preference. A broad fan creates more pairwise interpretations, especially when several decisions are mixed together. A narrow set of siblings keeps the parent relationship intelligible and gives the researcher a clearer basis for choosing the next parent. The workflow can still run siblings in parallel, but parallel execution does not remove the need to interpret each branch against the same decision and contract.
What the tree can and cannot guarantee
A well-shaped tree improves attribution, but it is not formal causal inference. Parentage does not prove that an improvement came only from the intended change. The code may contain hidden coupling; a metric may be unstable; the data may be inappropriate; or the run environment may differ in ways the branch does not record. The tree makes those questions easier to ask because it preserves the declared comparison, not because it answers them automatically.
The fixed-command rule has the same boundary. It prevents a common source of hidden variation, but it does not capture every package, dataset, random seed, accelerator behavior, or external service response. A branch can be correctly frozen and still fail to support a broad claim. Readers should treat the tree as an evidence scaffold: it defines what was intended to vary and what the workflow expects to hold steady, while scientific interpretation must examine whether those conditions were actually sufficient.
The tree is also not a substitute for a researcher's description of the hypothesis. A parent-child edge without a clear question can preserve technical ancestry while leaving the reasoning opaque. Conversely, a concise description connected to a fixed command and a recorded commit can make a small change legible without exposing every implementation detail. The goal is not to produce the largest history. It is to preserve the smallest chain that explains why the next experiment follows from the last one.
That chain changes the role of an agent. The agent may propose a variation, edit its branch, and prepare a run, but it should not silently rewrite an answered node or turn a launch failure into a new idea. OpenResearch gives the agent a tree in which those actions have different meanings. The next challenge is physical rather than conceptual: several agents need to explore different branches without sharing one mutable checkout. That is where private worktrees and session boundaries enter.