Reproduce “Robust-WAM: Bridging Generative Pretraining and Semantic Foresight in World-Action Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Haodong Yan, Junfeng Li, Junjie He, Zhide Zhong, MingMing Yu, Wenxuan Song, Jiaguan Zhu, Yangyang Zheng and 7 more
Robust-WAM improves robot control under visual shifts by adding future-scene semantic supervision to a video-generation world-action model while preserving its video-generation path. Learnable queries align action steps with semantic features of corresponding future frames. On LIBERO-Plus, GE-Act success rises from 78.0% to 80.9%, and clean LIBERO success from 96.5% to 97.3%; physical-robot evidence is limited to lighting shifts.
Key insights
- It is feasible to bridge the gap between powerful VAE-based generative pretraining and visual robustness by selectively injecting appearance-invariant semantic foresight into the action stream, rather than rebuilding models from scratch.
- Grounding robot actions in appearance-invariant future semantic information significantly improves out-of-distribution generalization capabilities without compromising performance on in-distribution tasks.
- The use of explicit temporal positional encoding for semantic query tokens is crucial for accurate temporal grounding, ensuring that the injected foresight correctly corresponds to specific future steps in the action plan.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.