Reproduce “From Scoring to Acting: Outcome-Verified Comparative Self-Distillation for LLM Agents” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xu Xia, Jinghua Piao, Min Yang, Xiaochong Lan, Jiaju Chen, Yong Li
Outcome-Verified Comparative Self-Distillation (OVCSD) is an on-policy framework that internalizes skill-free capabilities into LLM agents by transforming student failures into outcome-grounded comparative supervision. The approach improved success rates on ALFWorld by up to 37.5 percentage points and on WebShop by up to 24.2 percentage points over baselines, using minimal privileged interaction during training.
Key insights
- Failed student rollouts, even when uniformly unsuccessful, retain shared state structures and valid prefixes that can serve as effective and efficient intervention points for a teacher.
- Effective teacher supervision requires adaptive intervention from student-reached states and explicit environment verification of the subsequent successful task completion, not just local action preferences.
- The behavioral relationship between teacher and student trajectories fundamentally changes at the first state-aligned divergence, necessitating distinct learning objectives for localized error correction and subsequent task completion.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.