Reproduce “Fine-Grained Instruction-Guided Graph Reasoning for Vision-and-Language Navigation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yaohua Liu, Binkai Ou, Rong Fu, Amir H. Gandomi, Simon Fong
FIGR combines orientation-aware graph interaction with fine-grained instruction parsing for vision-and-language navigation. By separating visual and angular cues, modeling compositional landmarks and spatial relations, and aligning English, Hindi, and Telugu representations, it reaches 2.61 m navigation error, 76% success, and 67% SPL on R2R validation-unseen, while improving RxR validation-unseen performance over PRET to 75.1% success, 75.3 nDTW, and 64.8 sDTW.
Key insights
- Separating directional geometry from visual appearance improves graph representations: the full observation–graph design raises R2R validation-unseen SPL from 64.1 in the ablated setting to 67.0.
- Structured role and relation reasoning is especially useful for open-vocabulary and compositional instructions; removing fixed-vocabulary guidance reduces RxR out-of-vocabulary sDTW by 7.3 points relative to AOVG.
- Multilingual alignment provides larger gains for Hindi and Telugu than English, with sDTW improvements of 2.8 and 3.2 points respectively over using no adapter.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.