The Execution Gate: Can the Experiment Be Trusted?

MLR-Bench is designed to answer a practical question about AI research agents: what happens when a system must move beyond a plausible proposal and produce evidence that another person can inspect? Its benchmark separates idea generation, proposal formulation, experimentation, and paper writing so that readers can locate a failure instead of treating an end-to-end score as an unexplained verdict. The useful judgment is not simply which model scores highest, but which state the research has actually reached and what evidence is allowed to cross into the next state.

This matters because the exploration measures address the location of an idea, not the reality of its experiment. A promising direction still has to survive an execution environment, and a successful-looking run still has to support the interpretation in the paper. MLR-Bench makes those transitions visible through stepwise tests and then checks what happens when the stages are connected.

Turning open-ended research into observable stages

MLR-Bench contains 201 machine-learning research tasks sourced from workshops associated with ICLR, ICML, and NeurIPS over the preceding three years. The tasks cover nine broad topic categories, including large language and vision-language models, AI for science, trustworthy AI, machine-learning systems, computer vision, theory, reinforcement learning, multimodality, and emerging topics. They are open-ended enough to require a research proposal rather than a fixed-answer classification, but structured enough to give different agents the same starting challenge.

The benchmark uses two complementary modes. In stepwise evaluation, a model is tested on one research stage at a time. An idea-generation system receives a task and proposes an idea. A proposal-generation system receives the task and an idea and turns them into a more detailed plan. An experimentation agent receives a task, idea, and proposal, then works in an environment where its attempts and outputs can be recorded. A paper-writing model receives the experimental report, figures, and command log and must turn them into a paper. Each stage can therefore be judged on its own input and output boundary.

In end-to-end evaluation, the system receives a research task and is expected to produce the completed paper. That mode reflects the promise of autonomous research more directly, but it hides causality. If the final paper is weak, the score alone does not reveal whether the idea lacked novelty, the proposal was infeasible, the code failed, or the writing misrepresented an otherwise sound result. Stepwise evaluation supplies the missing checkpoints; end-to-end evaluation shows whether the checkpoints survive composition.

The denominator changes at the point where execution becomes expensive. Idea and proposal evaluations use all 201 tasks. The researchers select ten task/idea/proposal combinations for experimentation so that the runs remain manageable, and use the same ten-task scale for paper-writing and end-to-end comparisons. That distinction is part of the result, not a footnote. The broad upstream sample supports a wider view of how models formulate ideas. The downstream sample offers detailed evidence about execution, but its smaller size makes it less representative of the full task set.

This difference changes how an evaluator should read a headline result. A pattern across 201 tasks says that a capability appears across a varied benchmark collection, although it still depends on how those tasks were curated.