Reproduce “A Review of Online Diffusion Policy RL Algorithms for Scalable Robotic Control” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Wonhyeok Choi, Shutong Ding, Minwoo Choi, Jungwan Woo, Kyumin Hwang, Jaeyeul Kim, Ye Shi, Sunghoon Im
A comprehensive review and empirical analysis categorizes Online Diffusion Policy Reinforcement Learning (Online DPRL) algorithms into a structured taxonomy, evaluating their performance across 12 robotic tasks in a unified benchmark. The study reveals on-policy DPRL methods generally achieve higher performance ceilings in highly parallelized environments, whereas off-policy approaches offer greater robustness to limited parallelization and cross-embodiment variations.
Key insights
- On-policy Online DPRL algorithms, such as GenPO, achieve high performance in environments with extensive parallelization but degrade sharply when parallelization is limited.
- Off-policy Online DPRL algorithms, like DIPO, demonstrate robustness to limited parallelization and superior cross-embodiment generalization compared to on-policy methods.
- Backpropagation Through Time (BPTT)-based DPRL methods struggle with numerical instability and vanishing/exploding gradients with increased diffusion steps, unlike surrogate loss methods.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.