Reproduce “AbGen: Evaluating Large Language Models in Ablation Study Design and Evaluation for Scientific Research” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yilun Zhao, Weiyuan Chen, Zhijian Xu, Manasi Patwardhan, Yixin Liu, Chengye Wang, Lovekesh Vig, Arman Cohan
AbGen benchmarks the design of ablation studies by asking models to propose experiments for named components from research context. It includes 1,500 examples from 807 NLP papers. Models draft tests, but automated scores nearly tie leading models. DeepSeek-R1-0528 scores 4.11/5 on human evaluation, below the expert estimate of 4.80/5. These results assess design, not experiment execution.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.