Reproduce “SCOPE: Selective Critical-Step On-Policy Self-Distillation for Agentic Reinforcement Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Qirui Liu, Wei Cao, Yu Mi, Yichen Sun, Xuekun Zhang, Lei Liu, Yue Shen, Zhixuan Chu
SCOPE trains agents to attach corrective skills to selected error steps rather than apply one trajectory-level skill across all steps. It uses supervised initialization from offline annotations, then combines selective self-distillation with outcome-based reinforcement learning. At training step 150, Qwen2.5-3B-Instruct reaches 97.7% ALFWorld success and 85.2% WebShop exact success, though its WebShop normalized score trails SEED’s 88.5%.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.