Reproduce “GEMBA-MQM: Detecting Translation Quality Error Spans with GPT-4” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Tom Kocmi, Christian Federmann

Microsoft researchers developed GEMBA-MQM, a GPT-4-based method using fixed, language-agnostic three-shot prompting to achieve state-of-the-art reference-less machine translation quality estimation. The method provides granular error span detection and classification under the MQM framework, demonstrating 96.5% system-level pairwise accuracy on the WMT 2023 Metrics Shared Task.

Key insights

  • Large Language Models, particularly GPT-4, can achieve state-of-the-art performance in reference-less machine translation quality estimation, closely aligning with human judgments for system ranking.
  • Fixed, language-agnostic prompting, combined with a detailed error classification framework like MQM, enables scalable and diagnostically rich evaluation across diverse language pairs without requiring custom examples.
  • Fine-grained error categories within LLM prompts require careful validation, as models can misinterpret or misuse specific error types (e.g., "locale convention"), affecting overall accuracy and error distribution.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.