Reproduce “Efficient Online Reinforcement Learning for Diffusion Policy” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Haitong Ma, Tianyi Chen, Kai Wang, Na Li, Bo Dai
Researchers at Harvard University and Georgia Institute of Technology introduced Reweighted Score Matching (RSM) to efficiently train expressive diffusion policies for online reinforcement learning. Their algorithms, Diffusion Policy Mirror Descent (DPMD) and Soft Diffusion Actor-Critic (SDAC), achieved superior performance and computational efficiency on MuJoCo benchmarks, overcoming challenges of sampling from optimal policies.
Key insights
- Diffusion models can effectively learn and represent unnormalized probability distributions (energy-based models) that define optimal policies in frameworks like policy mirror descent and max-entropy reinforcement learning.
- Reweighted Score Matching (RSM) provides a theoretically sound mechanism to train score-based generative models by minimizing a weighted squared error, even when the target distribution cannot be directly sampled from.
- The design of specific reweighting functions in DPMD and SDAC enables efficient training in online RL by replacing intractable sampling from an optimal policy with tractable operations involving the current policy and forward diffusion kernels.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.