Reproduce “ETTRL: Balancing Exploration and Exploitation in LLM Test-Time Reinforcement Learning Via Entropy Mechanism” on OpenResearch

About this paper

Jia Liu, ChangYi He, YingQiao Lin, MingMin Yang, FeiYang Shen, ShaoGuo Liu

The ETTRL framework from Kuaishou Technology enhances Test-Time Reinforcement Learning for large language models in complex reasoning by introducing entropy-based mechanisms to reduce token consumption by 40% and improve learning stability, leading to up to a 69% relative performance gain on challenging mathematical benchmarks.

Key insights

• Intelligently branching at high-entropy tokens via ETMR significantly improves sampling diversity and pseudo-label accuracy while reducing computational cost by reusing shared prefixes.
• Reshaping advantage values with entropy (EAR) stabilizes policy optimization by preventing premature overfitting to noisy pseudo-labels, especially in early training stages.
• A balanced strategy for exploration during rollouts and robust learning via advantage reshaping is essential for efficient and effective test-time self-optimization of LLMs.

Summary by alphaXiv.