Reproduce “StructRL: Structured Action-Space Exploration for Flow-Based VLAs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jiarui Yang, Bin Zhu, Jingjing Chen, Na Zou, Yanwei Fu, Jianggang Zhu, Yu-Gang Jiang
StructRL introduces an action-space reinforcement learning framework for flow-based Vision-Language-Action models, relocating exploration stochasticity to the final action output and employing structured noise with temporal correlation and action-group-aware scaling. This approach enables more efficient online learning and improved out-of-distribution performance across various robotic manipulation benchmarks, achieving an 84% success rate on real-world "pick-banana" tasks.
Key insights
- Injecting structured noise within the intermediate steps of a flow-based generative model's denoising process results in dilution, rendering the noise less effective for exploration in the final action space.
- Placing policy stochasticity directly in the action space is crucial for preserving desired noise structures (temporal correlation, group-specific scaling) and significantly improves out-of-distribution performance in robotic tasks.
- The "last-step replay" mechanism provides a consistent and effective training signal for the flow decoder, bridging the gap between action-space exploration and likelihood-based RL objectives in generative models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.