Reproduce “Tree-Aware Posterior Distillation for Long-Horizon Agentic Reinforcement Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Heming Zou, Qi Wang, Yixiu Mao, Yun Qu, Yuyi Guo, Zeyuan Liu, Chenglin Li, Weijie Liu and 2 more
TreeOPSD trains long-horizon agents by branching continuations from shared histories, backing up verifier rewards into turn-level credits, and converting credits into bounded targets for policy updates. On Deep Research, it reaches a 21.20 exact-match average, versus 17.59 for TreePO, which uses the trees and credits but fixed-credit updates. This comparison covers the reported setup, not general convergence guarantees.
Key insights
- Tree-based branching supplies intermediate credit from terminal outcomes, while shared prefixes are scored once rather than counted repeatedly for every descendant trajectory.
- The distillation update matches the tree policy-gradient direction at the behavior policy but has a finite destination as the student changes across optimizer blocks.
- Pointwise credits and the affine target mapping performed better than the tested listwise-credit and logistic-mapping alternatives; the findings remain tied to the sampled trees and verifier rewards.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.