Reproduce “Policy of Thoughts: Scaling Test-Time Training for LLM Reasoning via Online Policy Evolution” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhengbo Jiao, Hongyu Xian, Qinglong Wang, Yunpu Ma, Zhebo Wang, Zifan Zhang, Dezhang Kong, Meng Han
PoT adapts a model’s policy during each coding problem, using execution feedback from failed attempts to guide later search. It combines Monte Carlo Tree Search with Group Relative Policy Optimization to update a temporary LoRA adapter while keeping the base model frozen. Across five code benchmarks, it averages 64.02 pass@1 versus 58.79 for TTT-Discover, with evidence limited to coding.
Key insights
- Dynamic, per-instance policy adaptation through gradient-based updates on transient LoRA adapters is crucial for improving LLM reasoning, outperforming static search methods.
- The integration of structured exploration (MCTS) with gradient-based policy internalization (GRPO) effectively converts external execution feedback into internal policy refinement.
- Smaller open-source models, when equipped with efficient test-time training like PoT, can achieve competitive or superior reasoning performance compared to much larger commercial LLMs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.