Reproduce “Weak-to-Strong Generalization with Failure Trajectories: A Tree-based Approach to Elicit Optimal Policy in Strong Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ruimeng Ye, Zihan Wang, Yang Xiao, Zinan Ling, Manling Li, Bo Hui
The authors organize weak-agent interaction records into trajectory trees and use shared-prefix branches for preference training or Monte Carlo Tree Search (MCTS) to train stronger agents. The method extends weak-to-strong generalization to multi-step tasks. For Llama-2-13B trained on weak Llama-2-7B trajectories, MCTS reaches ScienceWorld average reward 58.2, above SFT’s 53.6, but not the expert-trajectory model on every task.
Key insights
- Weak-to-Strong Generalization is a viable and effective paradigm for improving LLM agents in complex, interactive decision-making tasks, extending its applicability beyond simple classification.
- Explicitly structuring diverse trajectories (including failures) into a hierarchical 'trajectory tree' provides richer, more informative training signals than unstructured preference pairs, enhancing knowledge transfer from weak to strong models.
- Integrating offline Monte Carlo Tree Search (MCTS) with weak model-generated trajectory trees can effectively distill high-quality policies, enabling strong models to learn optimal behaviors and sometimes surpass expert-supervised performance without additional human feedback.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.