Reproduce “WMT24++: Expanding the Language Coverage of WMT24 to 55 Languages & Dialects” on OpenResearch

About this paper

Daniel Deutsch, Eleftheria Briakou, Isaac Caswell, Mara Finkelstein, Rebecca Galor, Juraj Juraska, Geza Kovacs, Alison Lui and 9 more

The WMT24++ project extends the WMT24 machine translation benchmark to include 55 languages and dialects, providing new human reference and post-edit data across diverse domains. This work benchmarks frontier large language models against traditional MT systems, observing that LLMs achieve higher scores on automatic metrics than human translations, though the authors caution against interpreting this as true "super-human" performance due to metric biases.

Key insights

• Post-edited human translations consistently achieved equal or higher quality scores compared to original human references across most languages, validating the efficacy of the post-editing process.
• Frontier LLMs demonstrated superior performance over both human translations and traditional MT systems when evaluated using automatic metrics like MetricX-24, consistently ranking highest across nearly all 55 languages.
• Automatic evaluation metrics are inherently biased against human translations, systematically rating them lower than human evaluators, which necessitates caution when interpreting claims of "super-human" machine translation quality.

Summary by alphaXiv.