Reproduce “SmoothRL: Online Reinforcement Learning During Asynchronous Execution” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Guang Gao, Yuxuan Nong, Baifu Huang, Jianan Wang
SmoothRL trains robot policies during asynchronous inference and execution, updating value gradients only through newly generated actions the robot executes. The critic also conditions on committed actions already in flight. On the robot, dynamic tossing success rose from 39% to 94% after 250 rollout episodes; evaluations used fine-tuning configurations and came from one online run.
Key insights
- Aligning the online RL optimization objective with the actual asynchronous execution process is critical for effective and unbiased policy adaptation in real-world robotics.
- Truncating value gradients to only the executed action segment prevents 'gradient contamination' and allows direct value-gradient backpropagation to policy parameters in asynchronous settings.
- Embedding the full asynchronous inference loop within the training rollouts ensures that the policy learns from data generated under the same timing constraints and action-execution mismatches encountered during deployment.
- Direct online learning in raw action space facilitates natural human intervention (especially residual corrections), enhancing learning efficiency and robot performance without complex latent space inversions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.