Reproduce “DiQAD: A Benchmark Dataset for End-to-End Open-domain Dialogue Assessment” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yukun Zhao, Lingyong Yan, Weiwei Sun, Chong Meng, Shuaiqiang Wang, Zhicong Cheng, Zhaochun Ren, Dawei Yin
Researchers from Baidu Inc., Shandong University, and Leiden University introduce DiQAD, a large-scale benchmark dataset comprising 100,000 real user-expert dialogues from an online platform, designed for end-to-end open-domain dialogue quality assessment. The dataset features a 3-scale human-epistemic quality score across six dimensions, enabling fine-tuned pre-trained Transformer models to achieve competitive assessment performance (e.g., ERNIE at 69.94% accuracy) while highlighting the limitations of current out-of-the-box large language models for this task.
Key insights
- Pre-trained Transformer models, such as Baidu's ERNIE, demonstrate superior performance in end-to-end dialogue quality assessment compared to classical and general LLM approaches.
- Large Language Models, when used in few-shot or Chain-of-Thought prompting setups, are less effective at discriminating nuanced human-epistemic dialogue quality on DiQAD without extensive fine-tuning.
- The dataset enables training models that exhibit good cross-domain generalizability, learning universal features of dialogue quality rather than domain-specific patterns.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.