Reproduce “Stable and Efficient Real-World Online VLA Post-Training via Asynchronous Replay-Anchored Policy Improvement” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jiarui Yang, Jiajin Zhang, Bin Zhu, Jingjing Chen, Yu-Gang Jiang
RAPolicy lets a robot collect rollout data while a learner updates a large vision-language-action policy, using replay-anchored critic targets and stored-latent actor updates tied to recorded actions. Starting from policies fine-tuned on 10 demonstrations per task, its four single-task evaluations average 86.3% success. In a five-task joint evaluation, success rises from 52% to 88%, versus 70% for HG-DAgger.
Key insights
- Replay anchoring addresses both sides of policy improvement: value estimates remain tied to observed actions, and actor updates increase the likelihood of higher-advantage recorded behavior rather than relying on Q-gradients for newly generated actions.
- Stored rollout latents preserve the correspondence between a flow-based VLA’s stochastic generation process and the replayed action chunk, providing more direct supervision than recomputed action targets.
- The combination of asynchronous execution and replay-constrained learning allows failures and human corrections to remain useful while emphasizing successful behavior through advantage weighting.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.