A Two-Gate Model for Trustworthy Research Agents
AI research agents need to clear two different boundaries before their work can be treated as scientific progress. The first asks whether a proposed direction broadens the space of questions and methods beyond its starting literature. The second asks whether the experiment behind a claim actually ran and produced evidence that can be inspected. The practical judgment is therefore not whether an agent is “good at research” in the abstract, but which kind of claim its evidence can support.
The two supplied studies make this distinction necessary. One finds that current agents can generate coherent ideas at enormous scale while remaining more concentrated and closer to their starting literature than human follow-on research. The other finds that, when agents must execute machine-learning experiments, failures can be hidden by synthetic or placeholder results. Neither study directly tests a finished two-gate architecture. Together, however, they support a disciplined way to classify directions, evidence, and uncertainty without confusing a promising proposal with a verified discovery.
Two gates ask different questions
The exploration gate concerns the status of a direction before its experiment is trusted. Its unit of judgment is not one impressive idea but the relationship among a collection of ideas, their seed literature, later research, and plausible consequence. The exploration study measures breadth by the average semantic distance among ideas within a research area. It measures seed distance by comparing each idea with the centroid of five seed papers. It adds future-frontier alignment through next-year human research keywords and estimates potential impact from the citation performance of nearby human papers.
These measures do not form a single ladder from bad to good. They describe different coordinates. Breadth asks whether outputs spread across distinct directions. Seed distance asks whether they escape the immediate starting frame. Frontier alignment asks whether they connect with themes that later human research studies. Estimated impact asks whether the neighborhoods nearest to the ideas resemble areas where human work receives attention. An idea can be far from its seed without being relevant to a future frontier, or it can sit near a promising neighborhood without representing a broad distribution.
The reported pattern is consistent enough to matter and qualified enough to interpret carefully. AI-generated ideas have average breadth of 0.554, compared with 0.599 for human-authored papers from the same areas, a 7.5% lower spread. Their average distance from the seed literature is 0.322, compared with 0.410 for human follow-on papers that cite at least one seed paper. AI ideas cover 28.5% of next-year frontier keywords, compared with 36.5% for those follow-on papers. Their estimated potential-impact score is 0.387, compared with 0.492 for the human comparison. The breadth and distance patterns hold in 11 of 12 broad fields, with Mathematics reported as the exception; the potential-impact difference also holds in 11 of 12 fields.
The important conclusion is about distributions, not individual destiny. These results do not show that every AI idea is derivative, that no AI idea can matter, or that human research always travels farther. They show that, under the study’s definitions and historical comparison, current AI ideation tends toward local elaboration. An exploration gate should therefore ask an agent to state what kind of movement it has achieved. Is the collection broad? Has it moved beyond the seed centroid? Does it connect with a future direction? Does it introduce a different research question, or mainly a different method for an established question?
The question–method distinction makes the gate sharper. Only 10.5% of generated ideas contain research questions absent from the five seed papers, while 90.4% introduce methods not present in those papers. That pattern does not make method innovation unimportant. A better method can be a valuable contribution to an established problem. It does mean that method-level novelty should not be silently promoted to evidence that the research problem itself has expanded. The comparison is also field-dependent: engineering and natural-science ideas more often retain their questions and change their methods, while Sociology and Business show higher rates of new questions.
The execution gate begins after a direction has been selected for implementation. Its unit of judgment is the evidence chain connecting a proposal to a completed experiment and then to a claim. MLR-Bench separates idea generation, proposal formulation, experimentation, and paper writing. The idea and proposal are forecasts and plans. The execution stage adds what the environment returned, what the agent attempted, and what outputs were produced.