Reproduce “VITA-E: Natural Embodied Interaction with Concurrent Seeing, Hearing, Speaking, and Acting” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xiaoyu Liu, Chaoyou Fu, Chi Yan, Chu Wu, Haihan Gao, Yi-Fan Zhang, Shaoqi Dong, Cheng Qian and 10 more
VITA-E enables humanoid robots to engage in natural, concurrent interaction by allowing them to see, hear, speak, and act simultaneously and interruptibly. The system achieved a 100% success rate for speech interruption and emergency stops, and 93.3% for action switching, significantly enhancing human-robot collaboration.
Key insights
- A dual-model architecture significantly improves human-robot interaction by enabling parallel processing, allowing a robot to maintain an ongoing task while remaining receptive to new, potentially urgent, user inputs.
- By training a VLM to generate explicit system-level control tokens, the VLM can act as a direct controller for the robot's interaction state, bridging the gap between semantic understanding and behavioral execution.
- Targeted data augmentation with synthetic interruption and action switching scenarios is critical for fine-tuning VLMs to reliably perform real-time interactive behaviors, including emergency stops and task switching.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.