Reproduce “LIBERO-Occ: Evaluating and Improving Vision-Language-Action Models under Scene-Induced Occlusion via Viewpoint Imagination” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Taishan Li, Jiwen Zhang, Siyuan Wang, Xuanjing Huang, Zhongyu Wei
This paper introduces LIBERO-Occ, a new benchmark for evaluating Vision-Language-Action (VLA) models under scene-induced occlusion, a common real-world challenge. It proposes Viewpoint Imagination (VIM), a framework enabling VLA models to internally generate complementary views to infer hidden information, improving task success rates on occluded tasks by 7.95 percentage points over strong baselines.
Key insights
- Existing VLA models demonstrate a significant, often hidden, dependency on complete visual information, experiencing substantial performance degradation under scene-induced occlusion compared to non-occluded scenarios.
- Viewpoint imagination, when integrated into a unified autoregressive VLA model, effectively enables perception completion by internally inferring missing visual evidence from partially observable scenes.
- A two-stage training strategy, combining dedicated viewpoint imagination pretraining with joint optimization alongside action prediction, is critical for learning cross-view spatial correspondence and maintaining a functional visual-token generation interface.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.