Reproduce “ResearchBench: Benchmarking LLMs in Scientific Discovery via Inspiration-Based Task Decomposition” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yujie Liu, Zonglin Yang, Tong Xie, Jinjie Ni, Ben Gao, Yuqiang Li, Shixiang Tang, Wanli Ouyang and 2 more
Researchers from Fudan University and Shanghai Artificial Intelligence Laboratory, among others, introduce ResearchBench, a large-scale, multi-disciplinary benchmark to evaluate large language models' capabilities in scientific hypothesis generation. The benchmark decomposes scientific discovery into inspiration retrieval, hypothesis composition, and ranking, revealing LLMs' proficiency in finding latent cross-domain connections but struggles in synthesizing truly novel hypotheses and exhibiting ranking biases.
Key insights
- Scientific discovery can be effectively decomposed into inspiration retrieval, hypothesis composition, and ranking for computational evaluation and benchmarking.
- LLMs demonstrate a notable ability to retrieve 'out-of-distribution' inspirations, indicating their capacity to identify latent cross-domain knowledge associations.
- LLMs encounter bottlenecks in composing truly novel hypotheses, likely due to optimization objectives favoring high-likelihood outputs, and exhibit significant positional biases in hypothesis ranking tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.