Reproduce “NovBench: Evaluating Large Language Models on Academic Paper Novelty Assessment” on OpenResearch

About this paper

Wenqing Wu, Yi Zhao, Yuzhuo Wang, Siyou Li, Juexi Shao, Yunfei Long, Chengzhi Zhang

Researchers at Nanjing University of Science and Technology and Queen Mary University of London introduce NovBench, a large-scale benchmark and a four-dimensional evaluation framework designed to assess Large Language Models' (LLMs) ability to generate textual novelty evaluations for academic papers. Experiments revealed that general-purpose closed-source LLMs like GPT-4o often outperform specialized models, and a human evaluation validated the proposed automatic metrics, showing a positive correlation with human judgments.

Key insights

• General-purpose closed-source LLMs, particularly GPT-4o and Gemini-2.5-flash, generally demonstrate stronger performance across evaluation metrics compared to open-source and specialized LLMs.
• Prompting strategies exhibit trade-offs: zero-shot yields highest relevance, few-shot improves coverage and correctness by simulating human expressive patterns, and RAG enhances clarity but can reduce relevance by introducing distraction.
• Specialized LLMs, despite fine-tuning on peer review data, show only marginal performance advantages in novelty understanding and can suffer from significant instruction-following deficiencies.

Summary by alphaXiv.