Reproduce “Trust Region Masking for Long-Horizon LLM Reinforcement Learning” on OpenResearch

About this paper

Yingru Li, Jiacai Liu, Jiawei Xu, Yuxuan Tong, Ziniu Li, Qian Liu, Baoxiang Wang

Researchers from Fudan University and The Chinese University of Hong Kong, Shenzhen, developed new, tighter trust region bounds (scaling as O(T) or O(T^3/2)) for autoregressive Large Language Model (LLM) reinforcement learning, representing a 409-fold improvement over classical O(T^2) bounds. They also introduced Trust Region Masking (TRM), a sequence-level mechanism that empirically stabilizes LLM training and enhances performance on long-horizon tasks like mathematical reasoning by controlling off-policy mismatch.

Key insights

• Traditional O(T^2) trust region bounds are numerically vacuous for long-horizon LLM tasks, failing to provide meaningful performance guarantees for sequences of thousands of tokens.
• Robust trust region bounds for autoregressive generation fundamentally depend on controlling the maximum token-level divergence, a critical metric that token-level clipping methods do not effectively manage.
• Standard token-level PPO clipping can exacerbate training instability due to its asymmetric nature, allowing large gradients from numerically noisy or high-importance-ratio tokens.

Summary by alphaXiv.