Reproduce “Error Span Annotation: A Balanced Approach for Human Evaluation of Machine Translation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tom Kocmi, Vilém Zouhar, Eleftherios Avramidis, Roman Grundkiewicz, Marzena Karpinska, Maja Popović, Mrinmaya Sachan, Mariya Shmatova
Error Span Annotation (ESA) introduces a balanced human evaluation protocol for machine translation, instructing annotators to mark errors before assigning an overall quality score. This approach achieves system ranking capabilities comparable to the detailed MQM protocol while being significantly faster and more reliable, making high-quality evaluation more accessible.
Key insights
- Requiring annotators to mark error spans before assigning an overall score significantly enhances the robustness and consistency of their judgments compared to direct scoring alone.
- High system ranking accuracy can be achieved with a simpler error marking process (two severity levels) without the full complexity and cost of detailed error categorization (MQM).
- Streamlining the annotation task leads to higher intra-annotator and inter-annotator agreement, indicating that the ESA protocol is easier for annotators to perform consistently.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.