Reproduce “LIBERO-X: Robustness Litmus for Vision-Language-Action Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Guodong Wang, Chenkai Zhang, Qingjie Liu, Jinjin Zhang, Jiancheng Cai, Junjie Liu, Xinmin Liu
LIBERO-X presents a new benchmark for Vision-Language-Action (VLA) models, featuring a hierarchical evaluation protocol and a high-diversity training dataset to comprehensively assess robustness and generalization. Experiments with LIBERO-X reveal that current VLA models exhibit persistent limitations in spatial generalization, structural invariance, and multi-step reasoning.
Key insights
- Current VLA models struggle with spatial generalization, overfitting to specific configurations and degrading significantly when objects are placed in out-of-distribution positions.
- Models lack structural invariance, showing substantial performance drops when scene topology is reconstructed, even if semantic relationships are maintained.
- Long-horizon tasks critically limit VLA model performance, with success rates plummeting to near zero for tasks requiring three or more sequential steps.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.