Reproduce “Beyond English-Centric Training: How Reinforcement Learning Improves Cross-Lingual Reasoning in LLMs” on OpenResearch

About this paper

Shulin Huang, Yiran Ding, Junshu Pan, Yue Zhang

Reinforcement Learning (RL) consistently demonstrates superior cross-lingual reasoning generalization in large language models compared to Supervised Fine-Tuning (SFT). Counter-intuitively, RL training on non-English data yields better performance and generalization than English training, a finding not observed with SFT.

Key insights

• Linguistic inconsistency, where RL models use mixed or non-training languages in internal reasoning steps, paradoxically contributes to stronger cross-lingual generalization by leveraging pre-trained multilingual knowledge.
• Full RL, with its active exploration and iterative policy improvement, is crucial for learning generalizable reasoning strategies, outperforming off-policy rejection sampling.
• Models with stronger cross-lingual generalization exhibit minimal semantic drift from the base model's representations, suggesting that preserving pre-trained multilingual structures is key for effective transfer.

Summary by alphaXiv.