Reproduce “LLM-based NLG Evaluation: Current Status and Challenges” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Mingqi Gao, Xinyu Hu, Jie Ruan, Xiao Pu, Xiaojun Wan

This survey from Peking University provides a timely and comprehensive taxonomy of LLM-based Natural Language Generation (NLG) evaluation methods, categorizing them into four distinct types based on how Large Language Models (LLMs) are utilized. The work analyzes the characteristics, strengths, and weaknesses of each approach, highlighting their performance, flexibility, reproducibility, and cost trade-offs, and identifies open problems and future research directions in this rapidly evolving field.

Key insights

  • LLM-based NLG evaluation methods can be categorized into four types based on how LLMs are utilized: LLM-derived Metrics, Prompting LLMs, Fine-tuning LLMs, and Human-LLM Collaborative Evaluation.
  • Each category presents distinct trade-offs in flexibility, reproducibility, performance, and cost, with human-LLM collaboration often achieving the highest correlation with human judgments but at a higher cost.
  • Despite their advancements, LLM-based evaluators still face challenges such as robustness issues, various biases (e.g., position, verbosity), concerns about reproducibility for proprietary models, and the need for unified benchmarks.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.