Reproduce “RSI-Forge: From Research Papers to Environments for Recursive Self-Improvement” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Renxiong Wang, Darvin Yi, Abril Herrlein, Anas Mahmoud, Advait Gosai, Lisiman Hua, MohammadHossein Rezaei, Xingang Guo and 11 more
RSI-Forge turns published papers into executable tasks through separate agents that construct, review, and independently reproduce each paper’s method as a baseline. Its corpus contains 210 environments across 18 fields. In 120 environments, at least one model improved score by 0.01 in 84%, and surpassed the reproduced baseline in 68; these are task results, not advances on original research problems.
Key insights
- Published papers can support a broad corpus of executable research tasks, but producing accepted environments still required a median of 170 million model tokens per environment, and 79% of the main-run environments needed repairs.
- Beating a reproduced baseline on an adapted task demonstrates task-level solution improvement, not necessarily scientific progress on the original paper’s problem or improved general research capability.
- Expert reviewers were more cautious than agent judges about shortcut resistance, saturation, and fidelity, indicating that automated checks and review do not eliminate the need for expert scrutiny.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.