Reproduce “Agents Are Systems, Not Models: Rethinking Agentic Evaluation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Luis Wiedmann, Leander Girrbach, Cordelia Schmid, Zeynep Akata
The paper evaluates how an agent’s configuration shapes performance across four scientific tasks. The authors vary information, reasoning, verification prompts, time budgets, and backbone models, repeating each configuration five times. Among genuine attempts on the three tasks where specialist references outperform backbones, run-to-run variation accounts for about 54% of score variance; information ranks first within this controlled study.
Key insights
- Agent performance depends on the full system configuration: task-specific information had the largest effect on score-gap closure in the three tasks where specialist models were expected to outperform the backbone.
- Configuration choices interact. More runtime can improve completion without improving scores, and is more useful when the agent has the information or capability needed to use that time effectively.
- Instructions alone rarely produce substantive verification; supplying an oracle scoring tool increased reference-based checking, though it also raised runtime and cost.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.