Reproduce “History Aware Multimodal Transformer for Vision-and-Language Navigation” on OpenResearch

About this paper

Shizhe Chen, Pierre-Louis Guhur, Cordelia Schmid, Ivan Laptev

A fully transformer-based architecture named HAMT is introduced for Vision-and-Language Navigation (VLN), which efficiently encodes text, long-horizon observation history, and current observations. This model achieved state-of-the-art performance across various VLN benchmarks, including a 3.2% relative improvement in Success Rate on R2R Test Unseen and a 9.5% relative gain in nDTW on R4R Val Unseen.

Key insights

• Explicitly modeling the entire navigation history using a hierarchical transformer (O(tK^2 + t^2) complexity) is crucial for long-horizon tasks and significantly outperforms recurrent or flat transformer approaches.
• A comprehensive suite of proxy tasks, particularly single-step action prediction/regression and spatial relationship prediction, enables robust pre-training and improves generalization before RL fine-tuning.
• End-to-end optimization of visual features from a pre-trained Vision Transformer, combined with RL+IL fine-tuning, further enhances navigation performance and generalizability to unseen environments.

Summary by alphaXiv.