Reproduce “Replicability Analysis for Natural Language Processing: Testing Significance with Multiple Datasets” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Rotem Dror, Gili Baumer, Marina Bogomolov, Roi Reichart
A framework for Replicability Analysis is introduced for Natural Language Processing (NLP), addressing the statistical multiplicity problem when evaluating algorithms across multiple datasets. The approach provides statistically sound methods for counting and identifying specific datasets where one algorithm truly outperforms another, ensuring more reliable experimental conclusions in NLP research.
Key insights
- Naive counting of significant per-dataset p-values in NLP evaluations is statistically flawed, leading to either underestimation of true effects in independent scenarios or severe overestimation in dependent ones.
- The appropriate statistical method for counting effects (Bonferroni for general dependency, Fisher for independence) is crucial for valid conclusions and depends on the assumed relationship between evaluation datasets.
- Adapting rigorous statistical frameworks from fields like psychology and genomics can significantly enhance the reliability and trustworthiness of experimental results in NLP.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.