Reproduce “MQM-Chat: Multidimensional Quality Metrics for Chat Translation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yunmeng Li, Jun Suzuki, Makoto Morishita, Kaori Abe, Kentaro Inui
A specialized evaluation metric, MQM-Chat, refines the Multidimensional Quality Metrics framework to accurately assess chat translation quality by incorporating seven error types, three of which are specific to informal communication. The framework revealed distinct strengths and weaknesses of large language models and commercial systems in handling chat complexities, particularly in Japanese and Chinese to English translations.
Key insights
- MQM-Chat effectively provides a granular diagnosis of chat translation quality, identifying specific error patterns, particularly those unique to informal communication like ambiguity resolution and dialogue consistency.
- Different machine translation models exhibit distinct strengths and weaknesses in handling chat content, with LLMs generally outperforming NMT models, but all models still struggle with specific chat-related issues.
- While automatic MQM-Chat annotation with LLMs shows promise for high-level system ranking, current few-shot prompting methods are insufficient for achieving human-level accuracy in fine-grained, span-level error detection and classification.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.