Reproduce “When Metrics Reward the Worst Translations: Internalizing Cultural Reasoning for Social Media Translation Evaluation” on OpenResearch

About this paper

Yiwen Qiu, Linjuan Wu, Dingming Li, Yizhou Liu, Zixuan Wang, Haolei Xu, Ye Guo, Daoxin Zhang and 2 more

Similarity-based translation metrics can reward literal Chinese–English social-media translations that humans judge as culturally wrong, with XCOMET scoring 29.36% of validation examples above 0.7 despite low human ratings. CuRIL uses gradually removed cultural hints and masked reinforcement learning to internalize this reasoning, raising Qwen3-8B’s four-class exact-match accuracy to 45.22% and restoring a monotonic relationship between predicted quality and error severity.

Key insights

• The central failure is cultural grounding rather than general model scale: Qwen3-235B achieved only 31.72% exact match and kappa 0.162, while the smaller CuRIL-trained Qwen3-8B reached 45.22% exact match and kappa 0.370.
• Providing cultural annotations at test time improved Qwen3-8B from 31.60% to 44.35% exact match, motivating a training strategy that transfers this information into the model instead of requiring retrieval or expert hints at deployment.
• Both components of the training design matter: removing hint decay lowered exact match to 40.29%, and removing gradient masking lowered it to 41.19%.

Summary by alphaXiv.