Reproduce “The Cancellation Hypothesis in Critic-Free RL: From Outcome Rewards to Token Credits” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tianhao Cheng, Zeyu Huang, Zihan Qiu, Yu Cheng, Edoardo Ponti, Yinghui Xu, Ivan Titov, Zenglin Xu
Researchers introduced the "cancellation hypothesis" to mechanistically explain how critic-free Reinforcement Learning (RL) methods achieve token-level credit assignment despite using only rollout-level rewards. This work demonstrates that opposing signals for common tokens in positive and negative rollouts tend to cancel out, allowing specific task-relevant tokens to be reinforced, and proposed batching interventions improved training stability and average pass@k scores on math reasoning benchmarks by up to 10.9%.
Key insights
- Critic-free RL updates exhibit "Token Flipping," where a significant portion of tokens are updated in opposition to their rollout's overall advantage sign, driven by interactions between positive and negative rollouts.
- Token-level credit assignment in critic-free RL is achieved through a sparse coupling mechanism, primarily affecting identical tokens predicted with low confidence within a batch.
- The "cancellation hypothesis" explains how common, non-discriminative token updates are mitigated by opposing signals, acting as an advantage-weighted filter that reinforces tokens more correlated with successful outcomes.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.