Reproduce “Visual Prompt Based Reasoning for Offroad Mapping using Multimodal LLMs” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Abdelmoamen Nasser, Yousef Baba'a, Murad Mebrahtu, Nadya Abdel Madjid, Jorge Dias, Majid Khonji

A zero-shot framework is introduced for off-road autonomous navigation, integrating SAM2-based segmentation with Vision-Language Models (VLMs) to identify drivable areas. The system achieves an average IoU of 0.6905 on real off-road datasets and demonstrates successful goal reachability in a custom simulation environment.

Key insights

  • Iterative point prompting for SAM2-based mask generation is significantly faster (approx. 3.5 seconds) than SAM2's built-in function, making it suitable for real-time off-road applications.
  • VLM performance in identifying drivable areas is sensitive to visual prompt format, the number of output masks (Single-Number Prompts generally superior), and linguistic prompt design, with optimal configurations varying by model size.
  • The generative and non-deterministic nature of some VLMs can lead to inconsistent navigation outcomes, indicating a need for fine-tuning VLM parameters like temperature and top-p constraints for increased reliability.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.