Reproduce “VLA-R: Vision-Language Action Retrieval toward Open-World End-to-End Autonomous Driving” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hyunki Seong, Seongwoo Moon, Hojin Ahn, Jehun Kang, David Hyunchul Shim
KAIST researchers developed VLA-R, a framework for open-world end-to-end autonomous driving, which integrates frozen vision-language models for perception with a vision-action retrieval mechanism. This approach enables robust robot navigation and exploration in unseen, unstructured environments, achieving high success rates and demonstrating action representation transferability across different robotic platforms with limited training data.
Key insights
- Formulating action generation as a retrieval problem significantly improves generalization and robustness in open-world autonomous driving compared to direct action prediction.
- Integrating frozen Vision-Language Models (VLMs) for open-world perception and aligning them with action representations via contrastive learning is effective for end-to-end control.
- A shared vision-language-action embedding space enables interpretable action selection and allows for transferable action representations across different robot kinematics.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.