Reproduce “Unifying Human and Statistical Evaluation for Natural Language Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tatsunori B. Hashimoto, Hugh Zhang, Percy Liang
Researchers from Stanford University developed Human Unified with Statistical Evaluation (HUSE), a framework that integrates human judgments of text typicality with model probabilities to provide a comprehensive metric for Natural Language Generation quality and diversity. This approach revealed that while methods like temperature annealing can improve perceived quality, they often lead to an overall degradation in performance by substantially reducing output diversity.
Key insights
- The optimal discriminator for distinguishing human from machine text relies solely on the two-dimensional sufficient statistic of reference and model probabilities, simplifying the theoretical basis.
- Human judgments of text typicality, aggregated from crowdworkers, reliably correlate with actual sentence frequencies and can effectively approximate the unknown reference probability `p_ref(y|x)`.
- The HUSE framework provides a diagnostic decomposition into quality (HUSE-Q) and diversity (HUSE-D) components, allowing for targeted analysis of model deficiencies.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.