Reproduce “Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarization” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Chenhui Shen, Liying Cheng, Xuan-Phi Nguyen, Yang You, Lidong Bing
Research by Shen et al. from Alibaba Group and National University of Singapore demonstrates that while large language models outperform traditional metrics in abstractive summarization evaluation, they do not yet achieve human-level reliability, exhibiting decreased consistency for higher-quality summaries and varying performance across different systems.
Key insights
- LLM evaluation reliability decreases as summary quality increases, with a strong negative meta-correlation observed for higher-quality summaries.
- LLM evaluation is candidate-dependent, meaning their alignment with human judgments varies significantly across different summarization systems, and also dimension-dependent.
- LLMs struggle to reliably differentiate between closely performing summarization systems, demonstrating lower accuracy on challenge sets of similar quality outputs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.