Reproduce “Exploring the Multilingual NLG Evaluation Abilities of LLM-Based Evaluators” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jiayi Chang, Mingqi Gao, Xinyu Hu, Xiaojun Wan
Researchers from Peking University systematically explored the multilingual evaluation abilities of LLM-based evaluators, demonstrating performance disparities across high and low-resource languages and investigating the impact of prompting strategies, model scale, and fine-tuning. The study found that excluding reference answers generally improves evaluation, larger models perform better, and fine-tuning can enhance multilingual capabilities with unexpected cross-lingual benefits from low-resource languages.
Key insights
- Including reference answers in prompts often detrimentally affects the evaluation performance of most LLM-based evaluators, challenging a common practice.
- Larger-parameter LLMs generally exhibit superior multilingual evaluation capabilities, but a significant imbalance exists, showing higher correlation with human judgments in high-resource languages.
- An LLM's highest sensitivity to specific perturbations often correlates with its most optimal overall evaluation performance in that language, suggesting a link between discerning subtle nuances and evaluation proficiency.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.