Reproduce “Learning an Efficient Multi-Turn Dialogue Evaluator from Multiple LLM Judges” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuqi Tang, Kehua Feng, Yunfeng Wang, Zhiwen Chen, Chengfei Lv, Gang Yu, Qiang Zhang, Keyan Ding and 1 more
Researchers developed MTDEval, a lightweight, open-source model for evaluating multi-turn dialogue quality, by distilling preference knowledge from multiple advanced LLM judges. The model demonstrates improved accuracy and inference efficiency compared to existing open-source baselines and competitive performance against proprietary LLMs across various evaluation tasks.
Key insights
- Aggregating knowledge from multiple diverse LLM judges significantly enhances evaluator robustness and generalization beyond what single-judge training can achieve.
- The effectiveness of a learned evaluator is strongly influenced by both the quantity and, more significantly, the quality of its constituent LLM judges.
- Encoder-based evaluators provide substantial inference efficiency gains by computing quality scores in a single forward pass, contrasting with the higher latency of generative LLM baselines.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.