Reproduce “SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositories” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ben Bogin, Kejuan Yang, Shashank Gupta, Kyle Richardson, Erin Bransom, Peter Clark, Ashish Sabharwal, Tushar Khot
The SUPER (Setting UP and Executing tasks from Research repositories) benchmark evaluates LLM agents on autonomously setting up and running experiments from less popular, "in-the-wild" research codebases. It reveals that state-of-the-art LLM agents achieve limited success on these end-to-end tasks, with GPT-4o reaching 16.3% accuracy, highlighting significant gaps in current capabilities.
Key insights
- Current state-of-the-art LLM agents are not yet capable of autonomously setting up and executing complex research experiments, demonstrating substantial limitations in comprehensive repository reasoning, flexible code editing, and dynamic error resolution.
- Agents frequently struggle with exploration of repository structure, hallucinating arguments for scripts or functions, and exhibiting inflexible strategies that hinder robust self-correction after encountering failures.
- While specialized tools like an explicit `edit` action and reflection mechanisms offer marginal improvements, they do not fundamentally overcome the core limitations in LLMs' intrinsic ability to resolve complex, multi-step issues.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.