Reproduce “DRAMA: Joint Risk Localization and Captioning in Driving” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Srikanth Malla, Chiho Choi, Isht Dwivedi, Joon Hee Choi, Jiachen Li

Researchers from Honda Research Institute, Samsung Semiconductor, and Stanford University introduced a new dataset and model for jointly identifying risky objects in driving scenes and generating natural language explanations for the perceived risk. The DRAMA dataset, comprising 17,785 interactive scenarios, enables the development of explainable AI that articulates what, where, and why a risk exists, demonstrating improved performance over single-task baselines.

Key insights

  • Incorporating optical flow (motion information) significantly improves the accuracy of risk captioning, underscoring the importance of dynamic cues in driving risk assessment.
  • Jointly training risk localization and natural language captioning, particularly by using the captioning module as a prior, mutually benefits both tasks, leading to better overall performance.
  • A learnable multi-task loss function is effective for balancing the optimization of correlated tasks like localization and captioning, outperforming manually weighted losses.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.