Reproduce “Measuring and Improving Semantic Diversity of Dialogue Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Seungju Han, Beomsu Kim, Buru Chang
Researchers from Seoul National University and Hyperconnect introduced Sem-Ent, a novel metric for measuring semantic diversity in dialogue generation that shows significantly higher correlation with human judgments compared to lexical metrics. They also developed DRESS, a learning method that improves semantic diversity by rebalancing the training objective, leading to generated responses that are more diverse and preferred by humans.
Key insights
- Semantic diversity is crucial for human perception of dialogue quality, and lexical diversity metrics are insufficient proxies, as demonstrated by Sem-Ent's strong correlation (Pearson 0.810) with human judgments.
- The semantic distribution of existing dialogue datasets is highly imbalanced, with a few 'head clusters' containing many generic responses, which models tend to overfit.
- Improving semantic diversity can be achieved without sacrificing coherency and can also lead to improvements in lexical diversity, as shown by DRESS's performance across multiple metrics.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.