Reproduce “Grounded Semantic Re-Binding for Robust Instruction Generalization in Vision-Language-Action Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zhaokai Yin, Zhipeng Zhang

Researchers from Shanghai Jiao Tong University identified that Vision-Language-Action (VLA) models fail with paraphrased instructions due to unstable joint vision-language encoding, not lost semantics. Their proposed architectural intervention, Grounded Semantic Re-binding (GSR), significantly improved paraphrastic invariance, boosting VLA-Adapter's success from 46.82% to 70.94% and SmolVLA's from 4.47% to 49.12% on the LIBERO-Para benchmark using only canonical data.

Key insights

  • VLA models retain the correct task identity internally even when failing on paraphrased instructions, indicating that the problem is not a complete loss of semantic understanding.
  • The root cause of paraphrastic failure is an architectural bottleneck where correctly encoded task semantics are mishandled by the action generation module due to unstable joint vision-language encoding.
  • The joint encoding of dynamic visual observations and text exacerbates feature variations, introducing systematic shifts in the feature space that make the downstream action policy vulnerable to wording changes.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.