The Missing Layer Between Intent and Behavior
For AI programming agents, an executable specification is the layer that connects what a person asks for with what a program actually does. This chapter focuses on how that layer operates: a candidate enters a runner or observer, selected behavior is exercised, and the resulting signal feeds a decision. By the end, readers should be able to tell whether a runnable artifact is checking correctness, equivalence, performance, regression behavior, or a deeper runtime property—and what each signal still leaves unknown.
The distinction matters because “specification” is often used for several different things. A prompt can describe a goal. A test can check an output. A formal contract can state a property with mathematical precision. These artifacts may all concern the same program, but they do not give an agent the same kind of information or impose the same cost. Treating them as interchangeable is one way to build an evaluator that sounds rigorous while measuring the wrong behavior.
From an intention to an observed signal
An executable specification is best understood as a procedure, not a paragraph. It accepts a candidate program, creates the conditions under which that program should operate, runs or observes it, and returns a signal that another part of the system can use. The signal may be binary—pass or fail—quantitative—such as a measured speedup—or structural, such as an observed runtime invariant. The procedure is the important part because it moves a claim about behavior from the agent’s description into an interaction with the program.
The sequence has several boundaries. First, a task description and its context define what the candidate is supposed to accomplish. The agent then generates or edits a program. The specification owns the next step: it supplies inputs, dependencies, setup, workloads, or instrumentation and invokes the candidate under those conditions. The runner owns the observations—outputs, exceptions, timings, traces, or state changes. Finally, a grader or controller converts those observations into a decision: accept, reject, repair, rank, or ask for human review.
That sequence also has failure paths. The candidate may not compile. The test harness may fail to construct a valid input. The program may run but violate an expected output relation. A timed workload may show that a “correct” optimization is slower. An observed property may be true on the executions collected so far but not generalize to unobserved states. An environment may be misconfigured, making an apparent program failure an artifact of the runner. A trustworthy system has to distinguish these cases rather than collapse every problem into a single negative score.
The boundary between observation and decision is especially important for an agent. A raw exception can tell the system that an execution failed, but it does not necessarily tell the model which assumption was wrong. A mismatch between candidate and reference outputs narrows the problem to a behavioral relation; a timing result narrows it to a workload and measurement; an invariant violation may point to a structural misunderstanding. The more diagnostic the signal, the more directly the next agent action can respond. Conversely, a single zero hides whether the failure came from code, setup, an invalid test, or an omitted case. Runnable does not automatically mean informative.
This is why an executable specification is an intermediate contract. Natural-language requirements are flexible enough to express purpose, priorities, exceptions, and human context. They are also ambiguous and difficult to verify automatically. Formal contracts offer stronger precision, but writing and checking them can require specialized knowledge and substantial infrastructure. A runnable check occupies a practical middle ground: it can make selected behavior observable at a scale and cost that an agent workflow can use, without pretending to prove every possible execution.
The distinction can be made explicit:
The diagram is not a hierarchy in which the bottom artifact always replaces the top one. Intent is still needed to say what matters. The executable layer needs an intended property to observe, and a formal contract may be unavailable or too expensive for the task. The useful question is which boundary the system needs to make visible. If a team cares about an API’s output on edge cases, a runnable test may be enough. If it cares about an optimization’s runtime under a representative workload, a pass/fail unit test is insufficient. If it cares about a pointer’s aliasing or nullability while translating a low-level program, the relevant evidence may be a dynamic observation rather than an output example.
A family of checks, not one magic test
The simplest executable specification is a fixed test: provide an input, run the candidate, and compare its output with an expected result. This is powerful when the input and property are clear, but a few fixed examples may leave large parts of the behavior unexamined. Input generators widen the view by producing random or adversarial cases, including boundary conditions that a hand-written example may miss. Their output still needs validation: generated inputs must be legal for the task, and their expected results must come from a trustworthy source.
This is where a reference implementation becomes useful. In ordinary test generation, the system may need to predict the correct output for every newly invented input. For many repository functions, that is harder than constructing the input itself. An equivalence harness takes a different route: it runs the original implementation and the candidate on the same input, then compares the relevant outputs or state changes. The original function becomes an operational oracle for the cases the harness exercises.
That arrangement changes the work required from the specification builder. Instead of writing a perfect expected answer for every complex object, the builder must create valid setup, choose inputs that reach meaningful behavior, preserve the dependencies the function expects, and decide which parts of the result count as equivalent. The harness is consequently closer to a small experimental environment than to a list of assertions. It owns the conditions of the comparison, while the reference implementation supplies the observed baseline. If either side is underspecified, the resulting pass signal can be precise about the wrong thing.
That design does not eliminate the need for judgment. The reference implementation supplies a baseline for the cases the harness exercises, but observational equivalence over a finite set of inputs is not the same as semantic equivalence over all inputs. The comparison may also need to understand structured results rather than compare two primitive values. A function that returns an object, mutates a buffer, writes a file, or depends on a custom data type may require setup and assertions that capture the behavior users actually care about. A harness can therefore be executable and still be a poor specification if it checks the wrong projection of the result.
Regression tests express a related but different relationship. They encode a behavior that was broken in one state and should be preserved or restored in another. A fail-to-pass test can identify whether an agent fixed a known issue; pass-to-pass tests can check that the fix did not damage existing behavior. Together they turn a natural-language bug report into a concrete transition: the old state fails a selected condition, the new state passes it, and unrelated behavior remains within the accepted boundary.
Performance harnesses extend the signal beyond binary correctness. A candidate must first produce the right result, then run a workload that measures time, throughput, memory, or another resource property. This matters because an agent can make a semantically correct change that is no faster, or a superficially faster change that only exploits the test’s narrow input pattern. A useful performance specification therefore pairs a correctness check with a workload and defines how to compare measurements. The result may be a speedup ratio or a threshold relative to a baseline rather than a simple pass/fail predicate.
Dynamic-analysis annotations provide yet another kind of evidence. They record properties that may be difficult to infer from source declarations alone: whether a pointer is ever null, how much memory an allocation backs, what runtime type a value has, or whether two arguments refer to the same object. Such observations are not ordinary input-output examples. They are structural facts about executions, and they can guide an agent before it commits to an implementation that must satisfy a stricter type or ownership system.
The choice among these forms should follow the failure surface. If the risk is a wrong answer, exercise inputs and compare outputs. If the risk is an interface misunderstanding inside a repository, preserve dependencies and compare candidate behavior with a known implementation. If the risk is an ineffective optimization, measure the workload after checking correctness. If the risk is an implicit runtime property, instrument the program and observe it. Adding more tests of the wrong kind may increase activity without increasing evidence.
The same contract can serve three different moments
The dissertation organizes these artifacts around three roles. In evaluation, an executable specification makes a task measurable. It defines the behavior against which candidates are compared, and it can expose differences that a static similarity metric misses. In training, the same kind of check can filter examples, label successful trajectories, define an environment, or become a reward. In inference, it can provide feedback for repair or help select one candidate from several sampled attempts.
The transition from one role to another is a change in what the signal does. During evaluation, the system asks, “How capable is this candidate under a controlled task?” During training, it asks, “Which experiences or outcomes should alter the policy?” During inference, it asks, “Which action or candidate should be chosen now?” The underlying test may be similar, but the acceptable latency, diagnostic detail, and tolerance for noise are different. A benchmark can afford to aggregate a result after a run; an agent in the middle of repair may need a failure explanation before it can decide what to edit next.
The roles are related but not identical. A benchmark grader may be designed for stable comparison across models. A training reward must be cheap and informative enough to support many iterations. An inference-time verifier must discriminate among candidates under a fixed compute budget. Reusing the same tests can be attractive because it preserves a common notion of correctness, but the system still has to account for cost, coverage, and the information each role needs. A sparse reward that is acceptable for a final training outcome may be too coarse to help an agent diagnose its next edit. A detailed diagnostic trace may be useful for repair but expensive to compute at benchmark scale.
This reuse also creates hidden coupling. If the training environments reward only what the evaluator later measures, an agent may become good at that measurement without acquiring the broader engineering capability the team wants. If a verifier uses generated tests that are easy to satisfy but weakly related to the issue, its ranking signal can steer selection in the wrong direction. If a performance target omits memory or maintainability, the agent may optimize the visible number while degrading the system around it. A specification is therefore part of the agent’s incentive structure, not a neutral window onto competence.
The practical rule is simple but demanding: name the behavior, choose an observation that is causally connected to it, and state the boundary of the decision. “The patch passed these tests” supports a claim about those tests. It does not automatically support a claim that the patch is safe in every environment, maintainable for every future engineer, or aligned with every project convention. “The workload ran faster” supports a claim about that workload and measurement procedure. It does not settle the trade-off with memory, portability, readability, or behavior outside the measured path.
This is why execution-backed checks remain empirical rather than becoming automatic proofs. They can make hidden behavior visible, but only through the inputs, environment, instrumentation, and comparison rule that someone selected. Their strength is operational: an agent can run them, learn from the result, and act on concrete evidence. Their weakness is also operational: a flawed harness can fail to run, fail to distinguish candidates, or reward a shortcut that satisfies the test without satisfying the intended system property.
The next step is to see what this model does to evaluation itself. Once a benchmark is built around live inputs, repository context, reference behavior, or timed workloads, its score no longer means simply “the model answered the prompt.” It means the candidate passed a particular executable account of what success was supposed to be. Understanding that account is the only reliable way to understand what the benchmark result does—and does not—say about a coding agent.