Reproduce “Planning Transformer: Long-Horizon Offline Reinforcement Learning with Planning Tokens” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Joseph Clinton, Robert Lieck
Planning Transformer adds sparse future-state predictions to a unified Decision Transformer, allowing action generation to be guided by an implicit plan without training separate hierarchical policies. The method improves performance most clearly on long-horizon AntMaze and FrankaKitchen tasks, while remaining competitive but less consistently superior on MuJoCo locomotion benchmarks.
Key insights
- Sparse future representations can shorten the effective decision horizon: the clearest gains occur in tasks requiring navigation over long distances or coordinated completion of multiple subtasks, while benefits are smaller on simpler locomotion problems.
- Representing plans as relative states is consistently better than using absolute states, and the choice of sampling strategy, goal encoding, action inclusion, and replanning interval materially affects performance.
- Predicted plans can provide qualitative diagnostics for failures, but the paper does not establish their interpretability through quantitative measures; performance also drops on the most difficult ultra-maze setting.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.