Reproduce “EchoRL: Reinforcement Learning via Rollout Echoing” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jinhe Bi, Aniri, Minglai Yang, Xingcheng Zhou, Wenke Huang, Sikuan Yan, Yujun Wang, Zixuan Cao and 4 more
EchoRL reuses successful model-generated answers when group-normalized rewards give every rollout zero advantage. It averages token entropy within reasoning steps, selects the highest-entropy step, and adds a next-token loss on that prefix to reinforcement-learning training. On Qwen2.5-Math-7B with GRPO, ID average rose from 44.5 to 47.0 and OOD from 55.6 to 58.4, though AIME24 and GPQA-Diamond declined in that comparison.
Key insights
- Even when all rollouts for a prompt are equally successful, they contain qualitatively different reasoning paths, and valuable "usable learning signals" systematically cluster around high-entropy decision points.
- Step-level entropy serves as a reliable proxy for critical learning signals, evidenced by expert trajectories exhibiting higher entropy and ablation studies showing that removing high-entropy steps significantly degrades accuracy.
- Reinforcing these high-entropy critical segments (EchoClips) through an auxiliary loss provides a dense, non-vanishing gradient that sustains learning and prevents entropy collapse, allowing for continuous performance improvement.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.