Reproduce “DRDT3: Diffusion-Refined Decision Test-Time Training Model” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xingshuai Huang, Di Wu, Benoit Boulet
DRDT3 unifies an improved Decision Transformer-style module with a conditional diffusion model to refine action predictions in offline reinforcement learning, overcoming Decision Transformer's limitations with suboptimal trajectories and lack of stitching ability. The framework demonstrates competitive or superior performance across D4RL benchmarks, yielding average improvements up to 41.9% over vanilla Decision Transformer on challenging tasks.
Key insights
- Incorporating Test-Time Training (TTT) layers into a Decision Transformer (DT) architecture (DT3) enhances the model's ability to learn from suboptimal trajectories and improves sequence modeling efficiency.
- Conditional diffusion models can effectively refine coarse action predictions from a DT-style module, leading to superior policy performance through iterative denoising.
- A unified training objective that balances a behavior cloning loss for coarse actions and a diffusion loss for refinement allows for synergistic optimization of both components, improving trajectory stitching.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.