Reproduce “SafeAuto: Knowledge-Enhanced Safe Autonomous Driving with Multimodal Foundation Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jiawei Zhang, Xuan Yang, Taiqi Wang, Yu Yao, Aleksandr Petiushko, Bo Li

SafeAuto introduces a framework for safer and more accurate autonomous driving by integrating a Position-Dependent Cross-Entropy (PDCE) loss for precise low-level control, a knowledge-enhanced post-safety verification module using Markov Logic Networks, and a multimodal retrieval-augmented generation (RAG) system. The approach reduced speed prediction RMSE by 5.8% and course prediction RMSE by 14.1% on BDD-X, and improved high-level behavior prediction accuracy by 13.0% on DriveLM compared to state-of-the-art baselines.

Key insights

  • A specialized loss function, Position-Dependent Cross-Entropy (PDCE), can adapt MLLMs for precise numerical regression of continuous control signals without sacrificing their high-level language capabilities.
  • Integrating explicit symbolic safety knowledge, such as traffic rules via Markov Logic Networks (MLNs), effectively mitigates MLLM hallucination and enforces compliance in safety-critical autonomous driving scenarios.
  • Multimodal Retrieval-Augmented Generation (RAG), especially when incorporating explicit environmental predicates, significantly enhances contextual understanding and decision-making by leveraging relevant past driving experiences.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.