Reproduce “Seeing Realism from Simulation: Efficient Video Transfer for Vision-Language-Action Data Augmentation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Chenyu Hui, Xiaodi Huang, Siyu Xu, Yunke Wang, Shan You, Fei Wang, Tao Huang, Chang Xu
The research from Shanghai Jiao Tong University and The University of Sydney introduces an efficient video augmentation framework that converts simulated Vision-Language-Action (VLA) videos into visually realistic training data, improving VLA model generalization from simulation to real-world tasks. This method boosts average success rates by 10.0% in challenging simulated scenarios and 13-15% on real robots, while reducing video generation time by 61.2%.
Key insights
- The denoising process in flow-based diffusion models exhibits distinct phases (rapid change, stable, adjustment), allowing for velocity caching to significantly reduce computational costs without compromising output quality.
- Enriching simulated data with diverse visual contexts, generated through conditional video transfer, effectively improves the generalization and robustness of VLA policies to real-world variations.
- Selectively augmenting a small subset of training trajectories that are both challenging and visually diverse, identified through coreset sampling, can yield substantial performance gains efficiently.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.