Reproduce “VERA: Explainable Video Anomaly Detection via Verbalized Learning of Vision-Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Muchao Ye, Weiyang Liu, Pan He
VERA introduces a framework for explainable video anomaly detection (VAD) that adapts frozen Vision-Language Models (VLMs) by verbally optimizing natural language guiding questions from coarsely labeled data. It achieves state-of-the-art performance on UCF-Crime (86.55% AUC) and XD-Violence (88.26% AUC) datasets without modifying VLM parameters or requiring external LLMs.
Key insights
- Frozen, pre-trained Vision-Language Models can be effectively adapted for complex, explainable VAD tasks through verbalized learning of guiding questions, circumventing the need for parameter modification or external reasoning modules.
- Verbalized learning, where an optimizer VLM iteratively refines natural language prompts based on performance feedback from a learner VLM, is a viable and efficient method for optimizing language-based parameters in multimodal tasks using coarsely labeled data.
- Decomposing abstract anomaly detection queries into specific, iteratively learned guiding questions significantly enhances a VLM's ability to reason about and detect anomalies, leading to substantial performance gains compared to generic prompts.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.