Reproduce “Learning from Unlabeled 3D Environments for Vision-and-Language Navigation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shizhe Chen, Pierre-Louis Guhur, Makarand Tapaswi, Cordelia Schmid, Ivan Laptev
Researchers at Inria, ENS, CNRS, PSL Research University, and IIIT Hyderabad introduce an automatic, scalable pipeline to generate large-scale Vision-and-Language Navigation (VLN) training data from unlabeled 3D environments. This approach creates HM3D-AutoVLN, a dataset an order of magnitude larger than previous ones, and improves the generalization of VLN models like DUET by over 7% SPL on unseen environments in datasets such as REVERIE and SOON.
Key insights
- Environmental diversity is more crucial for generalization than merely increasing the number of training samples within a limited set of environments.
- Joint training on both the automatically generated large-scale dataset (HM3D-AutoVLN) and a smaller downstream dataset is crucial for maximizing performance and preventing catastrophic forgetting.
- Models pretrained on HM3D-AutoVLN achieve strong performance even with limited supervised data, outperforming baselines with significantly less fine-tuning data, demonstrating data efficiency.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.