Reproduce “Recent Advances in Large Langauge Model Benchmarks against Data Contamination: From Static to Dynamic Evaluation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Simin Chen, Yiming Chen, Zexin Li, Yifan Jiang, Zhongwei Wan, Yixin He, Dezhi Ran, Tianle Gu and 3 more

This research provides a comprehensive survey of large language model (LLM) benchmarking, focusing on data contamination, and introduces a unified set of six criteria for evaluating dynamic benchmarks. It assesses existing static and dynamic evaluation methods, revealing the inherent limitations of static benchmarks and the need for standardized evaluation of dynamic approaches.

Key insights

  • Static benchmarks, even with enhancements, are fundamentally inadequate for reliably evaluating LLMs due to the pervasive issue of data contamination and the opacity of training data.
  • The current landscape of dynamic benchmarking lacks a consistent framework for self-evaluation, hindering systematic comparison and improvement of these methods.
  • No single existing dynamic benchmarking approach fully satisfies all optimal design principles, indicating inherent trade-offs across desired criteria like correctness, scalability, and diversity.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.