Overview

AutoGenBench makes Magentic-One comparisons more interpretable by isolating runs in fresh Docker containers, centralizing logs, and supporting repetition. The reported GAIA, AssistantBench, and WebArena scores describe particular models, prompts, tools, and protocols, not a universal ranking or production guarantee. Ablations suggest that structured orchestration and specialized workers both matter; error codes expose recurring problems in progress, verification, navigation, communication, and resource use. Read the percentages with task splits, exact versus softer measures, model refusals, sampling, and missing cost or latency evidence in view.

Learning outcomes

A benchmark score is meaningful only with its model configuration, task distribution, protocol, and verification arrangement.
Magentic-One’s ablations suggest value from both structured orchestration and specialized capability coverage, not simply from adding agents.
Failure codes diagnose recurring patterns, but language-model-assisted labels and sampled logs do not prove cause or transfer to deployment.

Keywords

AutoGenAutoGenBenchMagentic-Onebenchmarksablationsfailures

What the Evaluations Can—and Cannot—Tell Us

Magentic-One’s reported scores are useful only when read together with the conditions that produced them. This chapter focuses on that connection: how AutoGenBench controls stateful environments, what the benchmark results measure, and how ablations and failure analysis reveal the contributions and weaknesses behind a headline percentage. The reader will be able to report the main findings without turning them into an absolute ranking, and to distinguish evidence about orchestration from evidence about capability coverage or production reliability.

An evaluation is a controlled environment, not just a score

Agentic systems act on stateful environments, so the order and conditions of an experiment can change the result. If the first system installs a library, later systems may inherit it. If an agent deletes a file or leaves an environment in a broken state, subsequent tasks may be harmed. An apparently higher score can therefore reflect a cleaner starting state or a side effect left by an earlier run rather than a better workflow.

AutoGenBench was introduced to address this problem. It starts tasks in freshly initialized Docker containers, keeps the task logs in a central location, and supports repeated runs and parallel execution. Fresh containers reduce cross-run contamination. Repetition exposes variance from stochastic language-model calls. Central logs make the trace available for analysis, while isolation limits the consequences of an agent’s actions. These are not peripheral conveniences: they define what it means for two agent systems to face comparable conditions.

erDiagram CONFIGURATION ||--o{ RUN : tested_in BENCHMARK ||--o{ RUN : supplies_tasks PROTOCOL ||--o{ RUN : constrains RUN ||--o{ METRIC : produces RUN ||--o{ ABLATION : compares RUN ||--o{ ERROR_CODE : reveals METRIC }o--|| CLAIM : qualifies ABLATION }o--|| CLAIM : tests ERROR_CODE }o--|| CLAIM : limits

The diagram captures an evidence chain. A configuration is tested on a benchmark under a protocol; the run produces metrics, ablations, and observed errors; those outputs support claims with a defined scope. A score detached from the configuration or protocol is not a complete finding. Nor is an error label by itself a causal explanation. The relationship among them is what allows an evaluator to ask what changed and what that change can reasonably establish.

Magentic-One was evaluated on three demanding benchmarks. GAIA contains multimodal, real-world question-and-answer tasks that require several tools and steps. AssistantBench contains realistic, time-consuming questions involving web navigation and multi-step reasoning. WebArena places agents in functional synthetic websites and evaluates both answers and desired website states. These benchmarks differ in their task distributions and verification arrangements, so a percentage from one cannot be treated as directly interchangeable with a percentage from another.

The paper reports the results for two model configurations. With GPT-4o used across the team, Magentic-One achieved a 32.33% exact task-completion rate on the GAIA test set, 11.0% exact match and 25.3% softer accuracy on AssistantBench, and 32.8% on WebArena. With GPT-4o and o1-preview combined for selected roles, the reported rates were 38.0% on GAIA, 13.3% exact match and 27.7% softer accuracy on AssistantBench.

Published by DeepMox

FAQ

What does AutoGenBench control in an evaluation?

AutoGenBench controls stateful evaluation conditions by starting tasks in fresh Docker containers, keeping central logs, and supporting repeated or parallel runs. These measures reduce cross-run contamination and expose stochastic variation, but they do not make the resulting score a complete measure of user value.

Why can’t Magentic-One scores across benchmarks be compared directly?

Magentic-One scores across GAIA, AssistantBench, and WebArena cannot be compared directly because the benchmarks use different task distributions and verification arrangements. On AssistantBench, exact match and softer accuracy also measure different outcomes, so their percentages should not be collapsed into one result.

What do Magentic-One ablations reveal?

Magentic-One’s ablations suggest that structured orchestration and worker capabilities both contribute: replacing the full Orchestrator reduced GAIA validation performance by 31%, while removing individual workers reduced it by 21% to 39%. These are team-level results under the reported setup, not universal values for each component.

Previous11 minAutoGen Magentic-One Architecture

Next10 minAutoGen Autonomy: Practical Limits

AutoGen Magentic-One Evaluation

What the Evaluations Can—and Cannot—Tell Us

An evaluation is a controlled environment, not just a score

FAQ