Reproduce “Understanding Dataset Difficulty with $\mathcal{V}$-Usable Information” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Kawin Ethayarajh, Yejin Choi, Swabha Swayamdipta
The authors use V-information to measure how much label information a predictive model family can use from inputs relative to a null-input model. For natural-language inference, they compare predictions on original and transformed examples. On SNLI entailment examples, BERT-base mean PVI is 1.188 bits originally and 1.130 after word shuffling, suggesting limited impact of word order in this benchmark setup.
Key insights
- V-usable information provides a robust, model-specific metric for dataset difficulty, correlating with performance and serving as a more sensitive indicator of overfitting than accuracy.
- Pointwise V-information offers consistent instance-level difficulty estimates across models and tasks, showing high correlation with human judgment and effectiveness in detecting mislabeled data.
- Systematic analysis using input transformations and token-level PVI reveals known annotation artifacts and significant lexical biases in widely-used NLP benchmarks like SNLI and DWMW17.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.