Reproduce “How to Select Datapoints for Efficient Human Evaluation of NLG Models?” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Vilém Zouhar, Peng Cui, Mrinmaya Sachan
Researchers from ETH Zurich developed methods for selecting the most informative data points for human evaluation of Natural Language Generation models, aiming to preserve model ranking accuracy while reducing costs. Their framework, incorporating both output-based and source-based selection, achieved comparable evaluation results on tasks like machine translation with approximately 30% less human annotation data.
Key insights
- Strategic data selection for human evaluation consistently outperforms random sampling, making evaluation more efficient and robust.
- Selectors based on output diversity and metric consistency (how well an item's metric score aligns with overall model ranking) are often the most effective for distinguishing NLG models.
- The efficacy of metric-based selectors is tied to the quality of the underlying automated metrics, whereas diversity-based selection shows greater robustness.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.