Reproduce “ST-Booster: An Iterative SpatioTemporal Perception Booster for Vision-and-Language Navigation in Continuous Environments” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Lu Yue, Dongliang Zhou, Liang Xie, Erwei Yin, Feitian Zhang
The ST-Booster framework introduces an iterative spatiotemporal perception booster for Vision-and-Language Navigation in Continuous Environments (VLN-CE). It integrates global topological and local grid maps through feature-level fusion, achieving 59% Success Rate (SR) and 50% Success rate weighted by Path Length (SPL) on the challenging VLN-CE test-unseen split, outperforming prior methods by 2% SR.
Key insights
- Feature-level, bidirectional fusion between global topological and local grid maps enables synergistic information exchange, compensating for individual map limitations and forming a more robust environmental representation.
- Explicitly guiding waypoint generation with instruction-relevant attention heatmaps enhances decision robustness and offers interpretability into the agent's reasoning process.
- An iterative refinement framework, combining masked language modeling and hybrid single action prediction during pretraining, effectively trains multi-granularity perception for VLN-CE.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.