When One Final Test Is Too Late

For AI programming agents, a final end-to-end test is often too delayed to guide a long task and too narrow to define what a human team considers reliable work. This chapter examines two extensions of executable evidence: per-step runtime checks for a large C-to-safe-Rust translation, and human-centered evaluation for the standards, quality, judgment, and collaboration that a passing patch does not capture. The practical question is where feedback must appear inside the task and which decisions should remain visible to a person.

The two extensions solve different problems. Local specifications reduce the distance between an agent’s current decision and the evidence that can correct it. Human-centered evaluation expands the target beyond functional output. Together they show why reliability is not achieved by adding one final grader, one longer prompt, or one universal score.

Local evidence for a long translation

Translating a small function can sometimes be checked after the whole function is generated. Translating a multi-thousand-line library is different. An end-to-end test may reveal that the finished program diverges, but it does not necessarily identify which intermediate translation introduced the error or which assumption about a data structure was wrong. By the time the signal arrives, many later decisions may depend on the faulty one.

Syzygy addresses this granularity problem in a C-to-safe-Rust translation of Google’s Zopfli compression library. It decomposes the codebase into translation units and uses function-level equivalence tests as gates. A candidate Rust function is compiled, supplied with translated arguments, executed, and compared with the corresponding C behavior. If the comparison fails, the resulting difference can be used to repair that local translation before the error spreads through the rest of the dependency graph.

The reason local evidence is necessary is not simply that the codebase is large. C declarations leave important runtime facts implicit, while safe Rust requires those facts to become explicit choices. A C pointer may refer to one object, an array, or a moving position inside an allocation. It may be null, mutable, shared with another pointer, or associated with a value whose type is erased behind a generic pointer. The same surface declaration can therefore correspond to very different safe-Rust representations.

Static source text can describe how the pointer is declared and where it is used, but it may not settle the property that determines the correct representation. Syzygy mines observations from instrumented executions: the underlying type, the allocation size, whether null occurs, and whether two arguments alias. These observations are structural specifications. They do not merely add more example inputs; they expose runtime relationships that guide a type and ownership decision before the candidate is tested.

The distinction between structural and behavioral evidence is useful. A per-function equivalence test asks whether the translated function produces the same observable result on captured inputs. A dynamic annotation says something about the shape or relationship of the objects used during those executions. The first checks what the function did; the second helps the agent decide what kind of Rust program could safely and plausibly do it. One supports verification, the other reduces ambiguity before verification.

The local loop has a clear trigger.