Reproduce “Align-then-Slide: A complete evaluation framework for Ultra-Long Document-Level Machine Translation” on OpenResearch

About this paper

Jiaxin Guo, Daimeng Wei, Yuanchang Luo, Xiaoyu Chen, Zhanglin Wu, Huan Yang, Hengchao Shang, Zongyao Li and 3 more

Huawei Translation Services Center introduces "Align-then-Slide," a two-stage evaluation framework for ultra-long document-level machine translation from Large Language Models. The framework explicitly addresses source-target sentence misalignments, achieving a Pearson correlation of 0.929 with MQM scores on WMT 2020 Chinese-to-English and 0.943 on real-world datasets, significantly outperforming traditional metrics, and proving effective for guiding reinforcement learning in LLM-based MT.

Key insights

• Explicitly resolving sentence-level misalignments through dynamic programming is crucial for accurate document-level MT evaluation, as traditional metrics fail when one-to-one correspondence is broken.
• A multi-granularity (n-chunk) sliding window approach effectively captures contextual quality and mitigates evaluation inaccuracies arising from complex many-to-one source-target mappings.
• The framework's design is robust to choices of segmentation tools and underlying alignment models, consistently yielding high correlations with human judgment and providing an actionable signal for model optimization.

Summary by alphaXiv.