Reproduce “Inspire: Benchmarking Scientific Literature Search for Open Research Problems” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jianrong Ding, Zhengyan Shi, Jianyuan Zhong, Kai Qiu, Qi Dai, Yifan Yang, Chong Luo, Qiang Xu
INSPIRE evaluates agents searching redacted computer-science problems by measuring which graded cited antecedents they expose, select, and rank. Across 476 targets, Claude Opus 5 reaches 0.284 nDCG@10; it recovers at least one antecedent on 72.3% but at least three on 32.1%. Exposure shows the largest observed loss, while hindsight-SFT raises Qwen3.6-35B-A3B’s score from 0.082 to 0.166.
Key insights
- Literature search is a changing process: agents must form and update their own working notion of relevance rather than apply a criterion supplied in advance.
- Exposure is the largest measured loss point across evaluated systems; strong selection or ranking cannot recover useful papers that never appear in the search results.
- The benchmark’s graded references capture a target paper’s realized citation lineage, not every work that could scientifically help solve the problem.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.