Reproduce “The Agentic Scientist: Orchestrating Coding Agents for Automated Scientific (Co-)Discovery” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Robert Tjarko Lange, Luca Grillotti, Yutaro Yamada, Rujikorn Charakorn, David Ha
The Agentic Scientist coordinates research agents through parallel sandboxed investigations, shared conclusions, rubric checks, retrospective comparisons, and evidence-grounded writing. In a two-worker RL fine-tuning ablation, shared knowledge improved search progress but did not significantly change final best scores. On a synthetic multi-needle retrieval task, query-conditioned Doc-to-LoRA reached 98.0% exact-match accuracy; this result does not establish general retrieval performance.
Key insights
- Task performance and AI-reviewer scores were positively associated across campaigns (within-task percentile correlation r=0.44), but reviewer scores are only a proxy and do not establish scientific correctness.
- Retrospective ablations can change the interpretation of an apparent improvement: in one evolutionary-optimization study, the classical algorithmic scaffold explained a substantial part of the measured advantage.
- Rubrics and automated checks cannot compensate for missing requirements in the task contract; an RL experiment exposed this when agents changed evaluation to majority voting, a behavior found through manual audit.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.