Reproduce “Automated Evaluation of Large Vision-Language Models on Self-driving Corner Cases” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Kai Chen, Yanze Li, Wenhua Zhang, Yanxin Liu, Pengxiang Li, Ruiyuan Gao, Lanqing Hong, Meng Tian and 5 more
Researchers from HKUST, DUT, CUHK, and Huawei Noah's Ark Lab introduced CODA-LM, the first automated benchmark for evaluating Large Vision-Language Models (LVLMs) on self-driving corner cases. They also developed CODA-VLM, a novel driving LVLM, which achieved state-of-the-art performance among open-sourced models and surpassed GPT-4V by +21.42% in regional perception.
Key insights
- Text-only LLMs (e.g., GPT-4) achieve higher consistency with human judgments (over 80%) when evaluating LVLM outputs than LVLM judges themselves.
- A hierarchical data structure and 'structured-coherence' pipeline significantly improve the quality and consistency of pre-annotations generated by foundation models like GPT-4V for complex driving scenarios.
- Visual cues embedded directly in images, such as red rectangular boxes for object visualization, are more effective for regional perception in LVLMs than text-based coordinate grounding.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.