Reproduce “Bi-VLA: Bilateral Control-Based Imitation Learning via Vision-Language Fusion for Action Generation” on OpenResearch

About this paper

Masato Kobayashi, Thanpimon Buamanee

The Bi-VLA framework extends bilateral control-based imitation learning by integrating vision and natural language, allowing a single robot policy to execute multiple manipulation tasks. It achieved up to 90% success in language-disambiguated tasks and 75% in novel environments with distractors, demonstrating versatility and robustness through its CVAE-Transformer architecture with SigLIP language encoding.

Key insights

• Natural language input is crucial for disambiguating tasks where visual information alone is insufficient, enabling a single policy to perform context-dependent actions.
• The choice of language encoder impacts performance, with SigLIP demonstrating superior language grounding accuracy and execution reliability compared to DistilBERT.
• Integrating vision-language fusion into a force-aware bilateral control framework allows for multitask generalization and robustness in novel environments with distractors without degrading performance on vision-resolvable tasks.

Summary by alphaXiv.