Reproduce “ZeroVLN: Learning to Navigate without Supervised Fine-Tuning of the VLM Brain” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zixuan Huang, Yang Zhou, Guli Zhang, Sunzhu Li, Hongyan Xie, Zehao Chen, Fuzhen Zhuang, Kaixuan Wang and 3 more

ZeroVLN connects a vision-language model to continuous navigation by having its brain issue semantic movement requests to an action expert. The expert is supervised on trajectories grounded in those requests, while closed-loop reinforcement learning updates both policies without navigation-domain supervised fine-tuning of the brain. On R2R-CE, it reaches 52.7% success versus 46.5% for NaVILA, but has lower SPL, 29.2% versus 41.8%.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.