Reproduce “MetricX-24: The Google Submission to the WMT 2024 Metrics Shared Task” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Juraj Juraska, Daniel Deutsch, Mara Finkelstein, Markus Freitag

Google's MetricX-24, based on mT5-XXL, offers an enhanced machine translation evaluation metric by integrating a hybrid input mode and extensively augmented synthetic data. It achieves 94-100% accuracy in detecting specific translation errors and demonstrates segment-level accuracy gains of 1-2 points and system-level gains up to 11.5 points on the WMT23 MQM evaluation.

Key insights

  • Strategic synthetic data augmentation is highly effective in improving a metric's ability to detect and score common, yet challenging, translation failure modes like undertranslation and unrelated translations.
  • Combining noisy but abundant Direct Assessment (DA) data with high-quality but scarce Multidimensional Quality Metrics (MQM) data through a two-stage fine-tuning process allows for robust performance across many languages while maintaining sensitivity to subtle errors.
  • A hybrid input architecture enables a single evaluation model to flexibly operate as both a reference-based and reference-free (QE) metric, adapting its reliance on reference information for improved accuracy in diverse scenarios.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.