Reproduce “ST-VLA: Enabling 4D-Aware Spatiotemporal Understanding for General Robot Manipulation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
You Wu, Zixuan Chen, Cunxu Ou, Wenxuan Wang, Wenbo Huang, Lin Cao, Yangtao Chen, Weichao Qiu and 4 more
Researchers from Nanjing University and Huawei's Noah's Ark Lab introduce ST-VLA, a hierarchical framework that uses a unified 3D-4D spatiotemporal intermediate representation to enhance robot manipulation. The framework, powered by the ST-Human dataset and ST-VLM, improved zero-shot success rates in unseen tasks by 44.6% and robustness in cluttered scenes by 40.8%, enabling stable and generalized execution.
Key insights
- A unified 3D-4D intermediate representation serves as an effective bridge between high-level vision-language understanding and low-level continuous robot control, enhancing precision and stability.
- Explicitly training Vision-Language Models (VLMs) on large-scale, high-fidelity 3D-4D human manipulation data, such as ST-Human, enables them to acquire robust spatiotemporal reasoning capabilities essential for robot tasks.
- Hierarchical decomposition allows a high-level VLM to manage complex temporal reasoning and task decomposition, significantly improving the low-level policy's ability to execute long-horizon and generalized tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.