Reproduce “StereoVLA: Enhancing Vision-Language-Action Models with Stereo Vision” on OpenResearch

About this paper

Shengliang Deng, Mi Yan, Yixin Zheng, Jiayi Su, Wenhao Zhang, Yitao Zeng, Xiaoguang Zhao, Heming Cui and 2 more

StereoVLA uses stereo RGB views and auxiliary training tasks to connect geometric and semantic features with language-guided robot actions. Its GeoSem encoder combines FoundationStereo-derived geometry with left-image features, while depth and camera-parameter estimation encourage spatially aware representations. In large camera-view randomization, it achieves 53.9 ± 7.2% success versus 11.1 ± 4.6% for GraspVLA, under tested views that exclude the region directly behind the robot.

Key insights

• Jointly encoding both precise geometric information from stereo disparities and rich semantic features is crucial for improving VLA model performance in fine-grained robotic manipulation tasks.
• Co-training objectives, specifically Interaction-Region Depth Estimation focused on interaction areas and Camera Parameter Estimation, significantly enhance a VLA model's spatial reasoning and robustness to varying camera viewpoints.
• Leveraging intermediate feature representations (e.g., filtered cost volume) from a stereo depth foundation model offers a more efficient and geometrically rich source of information than full disparity maps for real-time VLA applications.
• Large-scale synthetic stereo datasets, combined with robust domain randomization, can effectively overcome real-world data scarcity and facilitate successful sim-to-real transfer for geometric understanding in embodied AI.

Summary by alphaXiv.