Reproduce “Any3D-VLA: Enhancing VLA Robustness via Diverse Point Clouds” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xianzhe Fan, Shengliang Deng, Xiaoyang Wu, Yuxiang Lu, Zhuoling Li, Mi Yan, Yujia Zhang, Zhizheng Zhang and 2 more
ANY3D-VLA is a framework that improves Vision-Language-Action (VLA) models by explicitly fusing diverse 3D point cloud representations with 2D visual inputs. The approach achieved an average success rate of 62.5% in challenging real-world zero-shot manipulation, representing a 29.2% improvement over the best baseline, and enhances robustness without requiring specialized depth hardware during inference.
Key insights
- Explicitly representing 3D geometry as compressed point clouds, fused with 2D representations, is more effective for fine-grained manipulation than implicit spatial priors or treating depth as an additional 2D channel.
- Training VLA models with diverse point cloud inputs (hybrid training) acts as an effective data augmentation strategy, significantly improving robustness and generalization across different depth qualities and reducing the sim-to-real gap.
- State-of-the-art model-estimated depth can often provide more accurate point clouds than commodity depth sensors, enabling VLA models to achieve robust performance without relying on expensive, specialized depth hardware.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.