Reproduce “Visual Cue Guided Video Planning for Generalizable Robot Navigation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Hojin Lee, Sizhe Lester Li, Maximilian Hilger, Susie Lu, Achim J. Lilienthal, Vincent Sitzmann, Daniel A. Duecker

CueNav uses a video planner with a bird’s-eye-view map of the robot and goal and a body-visible camera view. It predicts visual futures, whose dense optical flow an embodiment-specific inverse-dynamics model converts into continuous commands. In zero-shot 6×6 mazes, CueNav reached 55% of goals versus 30% without the map; in an unseen 1.0-meter passage, a 0.7-meter-wide Husky completed 70% of trials.

Key insights

  • Global route information can be supplied visually through a BEV cue: in unseen 6×6 mazes, adding the map raised success from 30% to 55% compared with an egocentric-only variant.
  • Visible body cues matter most in tight spaces, while the flow-based inverse-dynamics model is important for making executed motion match the generated visual plan.
  • A common video planner can be reused across wheeled and quadruped robots by separating visual planning from embodiment-specific action translation, though the cross-platform demonstration remains qualitative.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.