Reproduce “VAT: Vision Action Transformer by Unlocking Full Representation of ViT” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Wenhao Li, Chengwei Ma, Weixin Mao
The Vision Action Transformer (VAT) improves robot imitation learning by systematically utilizing the entire hierarchical representation trajectory of pre-trained Vision Transformers. This method yields an average success rate of 98.15% on the LIBERO benchmark, demonstrating enhanced robotic policy performance through deep and progressive perception-action fusion.
Key insights
- Leveraging the full hierarchical representation trajectory of a ViT, rather than only the final layer, is critical for achieving robust and precise robot policy performance.
- Intermediate layers of ViTs contain crucial fine-grained visual and spatial information that is often lost in the highly abstract final layer, yet is essential for complex manipulation tasks.
- A dedicated parameter space for action token processing via separate action modules facilitates deep perception-action fusion while preserving the integrity and powerful representations of pre-trained visual backbones.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.