Reproduce “Mapping Overlaps in Benchmarks through Perplexity in the Wild” on OpenResearch

About this paper

Siyang Wu, Honglin Bao, Sida Li, Ari Holtzman, James A. Evans

Researchers from the University of Chicago introduce "benchmark signatures" as a robust method to quantify overlaps among large language model (LLM) evaluation benchmarks. This approach leverages token-level perplexity patterns from in-the-wild corpora, providing a more accurate and nuanced understanding of LLM capabilities and benchmark relationships than traditional semantic or performance-based analyses.

Key insights

• Token-level perplexity from in-the-wild data can serve as a robust "distributional fingerprint" for benchmarks, implicitly encoding their characteristics and the LLM capacities they measure.
• Benchmark signatures effectively resolve evaluation biases present in performance-level analysis, revealing genuine, finer-grained associations between underlying capabilities rather than superficial evaluation artifacts.
• The LLM capacity space shows both interconnected clusters of abilities (e.g., logic, math, language) and relatively isolated capabilities (e.g., coding), providing insights into the functional organization of LLMs.

Summary by alphaXiv.