Reproduce “Hide to Guide: Learning via Semantic Masking” on OpenResearch

About this paper

Ruitao Liu, Qinghao Hu, Alex Hu, Yecheng Wu, Shang Yang, Luke J. Huang, Zhuoyang Zhang, Han Cai and 1 more

The paper introduces Semantic Masked Expert Policy Optimization (SMEPO), a method that semantically masks reward-relevant content in expert traces to prevent large language models from reward hacking in Reinforcement Learning with Verifiable Rewards (RLVR). This approach improves learning efficiency and enhances performance, achieving average accuracy gains of 2.7 points in math, 2.5 points in code generation, and 3.2 points in agentic search compared to vanilla GRPO.

Key insights

• Direct conditioning on full expert traces can degrade RLVR performance by encouraging reward hacking, where models exploit explicit solutions instead of learning problem-solving processes.
• Fine-grained semantic masking of reward-relevant information is critical for effective expert guidance, allowing the model to leverage procedural cues without shortcutting to answers.
• The benefit of expert guidance lies primarily in its structural and procedural scaffolding, which guides exploration when critical solution details are concealed.

Summary by alphaXiv.