Reproduce “DriveSOTIF: Advancing Perception SOTIF Through Multimodal Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shucheng Huang, Freda Shi, Chen Sun, Jiaming Zhong, Minghao Ning, Yufeng Yang, Yukun Lu, Hong Wang and 1 more
Researchers from the University of Waterloo and collaborators advanced autonomous driving safety by fine-tuning multimodal large language models (MLLMs) on a novel, LLM-generated dataset called DriveSOTIF. This approach improved MLLM performance in image captioning and visual question answering for perception-related Safety of the Intended Functional-Functionality (SOTIF) risks, achieving efficient real-time inference suitable for on-board deployment.
Key insights
- Fine-tuning MLLMs with SOTIF-specific data enables them to acquire human-like cognitive abilities for analyzing, reasoning about, and reacting to complex driving risks.
- Automated dataset generation using a multi-agent LLM framework can efficiently create high-quality, diverse, and contextually rich VQA and image captioning data for niche domains like SOTIF.
- Model quantization significantly reduces memory footprint and improves inference speed, making powerful MLLMs feasible for real-time, on-board deployment in resource-constrained autonomous vehicles.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.