Reproduce “PaSBench-Video: A Streaming Video Benchmark for Proactive Safety Warning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yusong Zhao, Yuejin Xie, Youliang Yuan, Junjie Hu, Jitian Guo, Yujiu Yang, Pinjia He
PaSBench-Video introduces a streaming video benchmark to evaluate Multimodal Large Language Models (MLLMs) for proactive safety warning. The benchmark reveals that current MLLMs struggle with temporal calibration and high false-positive rates, achieving a top `First-Act@1` score (correctly timed and content-accurate warning) of only 20.0%.
Key insights
- Current MLLMs largely function as simple threshold detectors, exhibiting a near-linear trade-off between recall and false-positive rates rather than demonstrating sophisticated risk reasoning.
- Model performance varies significantly across domains, indicating struggles with subtle, spatiotemporal cues (e.g., driving) compared to more overt, anomalous risks (e.g., daily life).
- Increased model sensitivity primarily shifts 'missed' warnings to 'too-early' warnings, revealing an inability to precisely identify the actionable onset of risk within the temporal window.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.