Reproduce “Reconstruction: A Blind Benchmark for Recovering Research Ideas from Pre-Publication Bibliographies” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shaolong Chen, Yanlin Fei, Nazhou Liu, Xinmiao Yu, Lei Li, Rahul Thapa, Madalina Ciobanu, Navan Preet Singh and 2 more
The authors test whether models can recover a paper’s central idea from earlier references alone. The seed paper and later literature are withheld; an independent judge compares five hypotheses with the seed. Across 643 papers in six domains, a four-model pipeline achieves a 35.6% match rate versus 14.6% for the strongest single-model baseline, comparing full pipelines rather than collaboration alone.
Key insights
- While individual frontier LLMs showed limited performance in reconstructing research ideas (Match rates from 3.4% to 15.0%), collaborative multi-agent systems significantly boosted recovery rates.
- The multi-agent approach, incorporating cross-model review and competitive selection, provides value beyond simply selecting the best performing individual outputs, indicating benefits from collaborative refinement.
- LLMs demonstrated a capacity for bibliography-conditioned inference even for papers published after their knowledge cutoffs, suggesting genuine reasoning capabilities beyond direct memorization of the seed paper.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.