Reproduce “Combining Optimal Control and Learning for Visual Navigation in Novel Environments” on OpenResearch

About this paper

Somil Bansal, Varun Tolani, Saurabh Gupta, Jitendra Malik, Claire Tomlin

LB-WayPtNav enables autonomous vision-based navigation in novel, cluttered indoor environments using only a monocular RGB camera. The framework combines learning-based waypoint prediction with model-based optimal control to generate smooth, dynamically feasible trajectories, demonstrating zero-shot transfer from simulation to real robots and achieving higher success rates than purely end-to-end learning and real-world depth-based mapping methods.

Key insights

• The hybrid approach effectively leverages learning for robust visual interpretation and generalization to novel environments, while model-based optimal control ensures dynamic feasibility, robustness, and safety during physical execution.
• Robust feedback control (LQR) is crucial for enabling zero-shot transfer from simulation to real hardware by effectively compensating for unmodeled dynamics, disturbances, and discrepancies between simulation and reality.
• Relying on monocular RGB vision for navigation can be more robust in real-world cluttered environments than using real-world depth sensors, which often fail due to issues with shiny, transparent, or thin objects, or ambient lighting.

Summary by alphaXiv.