Reproduce “Evaluating and Enhancing Large Language Models for Novelty Assessment in Scholarly Publications” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ethan Lin, Zhiyuan Peng, Yi Fang
Researchers from Santa Clara University introduce `SchNovel`, a benchmark for evaluating Large Language Models' ability to assess scholarly novelty within publications, and propose `RAG-Novelty`, a method that integrates retrieval-augmented generation to enhance this capability. Their evaluation shows `RAG-Novelty` improves accuracy over baselines, while also revealing LLMs' susceptibility to biases like prompt order and institutional affiliation when assessing novelty.
Key insights
- Scholarly novelty assessment requires dynamic access to evolving knowledge, making retrieval-augmented generation (RAG) crucial for LLMs.
- LLMs exhibit significant biases (e.g., position bias, affiliation bias) when evaluating scholarly novelty, which can impact fairness in automated academic processes.
- The inherent complexity of a research field and the temporal gap between papers significantly influence LLM performance in novelty assessment.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.