Reproduce “Benchmark Transparency: Measuring the Impact of Data on Evaluation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Venelin Kovatchev, Matthew Lease
A framework called "Benchmark Transparency" quantifies how data characteristics, such as ambiguity and difficulty, influence natural language processing model performance and rankings. This approach reveals that data distribution changes can affect evaluation outcomes more significantly than the choice of metrics, also allowing for out-of-distribution performance prediction.
Key insights
- Data distribution has a substantial impact on both absolute model performance and relative model rankings.
- Different data dimensions, such as difficulty and label noise, affect model performance to varying degrees, often more dramatically than dimensions like length or perplexity.
- Changes in data distribution can influence evaluation outcomes more significantly than the choice of evaluation metrics themselves.
- The six identified data dimensions (Ambiguity, Difficulty, Discriminability, Length, Noise, Perplexity) are largely orthogonal, capturing independent aspects of the data.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.