Reproduce “General Evaluation for Instruction Conditioned Navigation using Dynamic Time Warping” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Gabriel Ilharco, Vihan Jain, Alexander Ku, Eugene Ie, Jason Baldridge

Normalized Dynamic Time Warping (nDTW) and its success-weighted variant, SDTW, are proposed as novel evaluation metrics for instruction-conditioned navigation, showing superior correlation with human judgments of path fidelity and improving agent performance when used as reinforcement learning reward signals.

Key insights

  • Dynamic Time Warping (DTW) can be effectively normalized and adapted to measure trajectory fidelity in navigation tasks, offering soft penalties for deviations and preserving sensitivity to the sequence of waypoints.
  • nDTW and SDTW correlate substantially better with human perceptions of preferred path similarity compared to existing metrics, indicating they more accurately reflect human intuition about navigation correctness.
  • Employing nDTW as a reward signal for reinforcement learning agents leads to improved performance in instruction-conditioned navigation, yielding agents that follow instructed paths more faithfully while also enhancing traditional goal-oriented metrics.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.