Reproduce “PaperCompiler: Faithful Paper-to-Code Generation via Repository-Level Specification Compilation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yunhao Liu, Hong Phuc Pham, Jaehong Yoon
PaperCompiler introduces a specification-compilation pipeline for converting research papers into repository-level implementations while preserving method details, interfaces, and evaluation procedures. Across 90 papers, it improves over PaperCoder in reference-free, fine-grained, and reference-based assessments, with the largest gain in alignment to author repositories: 4.152 versus 3.647 on average.
Key insights
- The paper treats paper-to-code generation as an information-preservation problem: the central challenge is maintaining scientific constraints across transformations from evidence to architecture to implementation.
- Requirement reconciliation and file-level contracting are more effective for reference-based fidelity than simply providing broader context; removing them in ablations reduces scores from 4.38 to 3.86 and 3.92, respectively.
- Explicit uncertainty is preferable to silent approximation: unsupported modes and missing external resources should remain visible or fail clearly rather than be replaced with dummy implementations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.