Reproduce “SciImpact: A Multi-Dimensional, Multi-Field Benchmark for Scientific Impact Prediction” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hangxiao Zhu, Yuyu Zhang, Ping Nie, Yu Zhang
Researchers from Texas A&M University and collaborators developed SCIIMPACT, a multi-dimensional and multi-field benchmark designed to evaluate models' capabilities in predicting scientific impact beyond traditional citations. The benchmark, featuring contrastive paper pairs across 19 scientific fields and 7 impact dimensions, demonstrated that supervised fine-tuning significantly improves the performance of large language models, with SFT-Qwen3-4B achieving the highest average accuracy of 0.685.
Key insights
- Supervised fine-tuning on the SCIIMPACT benchmark substantially boosts LLM performance across all impact dimensions, enabling smaller open-source models to outperform larger, untuned counterparts.
- Different impact dimensions provide complementary supervision signals, as multi-dimension training yields higher overall accuracy than single-dimension training, validating the benchmark's multi-dimensional design.
- Prediction difficulty varies significantly across impact dimensions and scientific fields, with 'Award' recognition generally easier to predict and 'Computer Science' and 'Other Fields' proving more challenging, highlighting the heterogeneous nature of scientific impact.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.