Reproduce “Paper2Code: Automating Code Generation from Scientific Papers in Machine Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Minju Seo, Jinheon Baek, Seongyun Lee, Sung Ju Hwang
PaperCoder turns scientific papers into repositories by planning architecture and file-specific requirements, then generating code in dependency order without supplied code. On PaperBench Code-Dev, it scores 45.14 ± 0.3% with o3-mini-high, versus 16.4 ± 1.4% for IterativeAgent, averaged over three runs on 20 papers. This code-development rubric does not test execution or whether experimental results match.
Key insights
- A structured, multi-stage, multi-agent LLM architecture is effective for interpreting extensive, unstructured scientific documents and translating them into functional, repository-level code.
- Explicit planning for architectural and logical design, including inter-module dependencies and execution order, is crucial for generating coherent and complete codebases from high-level paper descriptions.
- Integrating self-refinement steps within the planning and analysis phases significantly improves the quality of their outputs and reduces downstream errors in code generation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.