Reproduce “DriVLMe: Enhancing LLM-based Autonomous Driving Agents with Embodied and Social Experiences” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yidong Huang, Jacob Sansom, Ziqiao Ma, Felix Gervits, Joyce Chai

Researchers at the University of Michigan's SLED Group developed DriVLMe, an LLM-based autonomous driving agent that integrates embodied experiences from simulated driving and social experiences from human-human dialogue. This agent aims to enable natural human-vehicle communication and navigation, achieving competitive performance on open-loop benchmarks like SDN and BDD-X while also revealing practical limitations in closed-loop human interaction.

Key insights

  • Integrating both embodied experiences (simulated driving environment) and social experiences (real human-human dialogue) is crucial for developing robust communicative autonomous driving agents.
  • Foundation models, specifically fine-tuned Vicuna-7B, can function as central reasoning engines for complex autonomous driving tasks, handling multimodal inputs, symbolic planning via tool-use, and situated dialogue management.
  • While open-loop benchmark evaluations demonstrate strong capabilities, closed-loop human-in-the-loop studies are essential for uncovering practical interaction challenges, such as real-time inference and nuanced visual understanding in dynamic environments.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.