Reproduce “EVA2.0: Investigating Open-Domain Chinese Dialogue Systems with Large-Scale Pre-Training” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuxian Gu, Jiaxin Wen, Hao Sun, Yi Song, Pei Ke, Chujie Zheng, Zheng Zhang, Jianzhu Yao and 3 more
A systematic investigation into crucial factors such as data quality, model architecture, training strategies, and decoding for open-domain Chinese dialogue systems led to EVA2.0, a state-of-the-art model that significantly outperforms existing baselines, with a smaller version achieving comparable performance to its larger predecessor.
Key insights
- Data quality is paramount, as the refined EVA2.0-Dataset (60GB) led to significantly higher model performance than the much larger, lower-quality WDC-Dialogue (181GB).
- Models with balanced encoder and decoder layers (e.g., 12-12) consistently performed best, while the introduction of pseudo-role information surprisingly degraded performance in social media dialogue contexts.
- Training a model from scratch on high-quality dialogue data proved generally more effective for building a chit-chat bot than further pre-training from a long-document general generative model, which can sacrifice conversational skills.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.