Reproduce “BLEURT Has Universal Translations: An Analysis of Automatic Metrics by Minimum Risk Training” on OpenResearch

About this paper

Yiming Yan, Tao Wang, Chengqi Zhao, Shujian Huang, Jiajun Chen, Mingxuan Wang

Researchers from Nanjing University and ByteDance AI Lab uncovered critical robustness defects in popular automatic machine translation metrics, including BLEURT and BARTScore, when guiding model training via Minimum Risk Training. They demonstrated that models optimized with these metrics could produce "universal adversarial translations"—high-scoring, low-quality outputs—and proposed strategies like combining NLL loss or employing metric ensembles to mitigate these issues and improve translation quality.

Key insights

• Optimizing NMT models with certain metrics (e.g., BLEURT, BARTScore) via MRT can lead to catastrophic training collapse, where models generate high-scoring but semantically flawed "universal adversarial translations."
• These robustness defects are caused by distribution biases in MT training corpora (e.g., frequent semantic patterns) and the inherent tendencies of metric modeling paradigms (e.g., BARTScore's generability bias, BLEURT's susceptibility to learned corpus biases).
• Robust metrics like COMET and UniTE consistently guide stable training, often improving other metrics, whereas non-robust ones cause performance degradation across the board, highlighting fundamental differences in their evaluation criteria.

Summary by alphaXiv.