Reproduce “VidHalluc: Evaluating Temporal Hallucinations in Multimodal Large Language Models for Video Understanding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Chaoyu Li, Eun Woo Im, Pooyan Fazli
Researchers from Arizona State University introduce VIDHALLUC, a large-scale benchmark for evaluating temporal hallucinations in Multimodal Large Language Models (MLLMs) for video understanding, and DINO-HEAL, a training-free mitigation method. The work identifies widespread hallucination issues in current MLLMs across action, temporal sequence, and scene transition understanding, while DINO-HEAL improves MLLM robustness by an average of 3.02% by reweighting visual features with DINOv2 saliency maps.
Key insights
- MLLMs consistently exhibit high vulnerability to temporal hallucinations, performing significantly worse on adversarial binary action questions and struggling to accurately interpret the chronological order of events.
- Hallucinations are partly attributed to the inductive bias of visual encoders (e.g., CLIP), which prioritize global contextual scenes, leading to an insufficient differentiation of fine-grained visual details needed for accurate temporal reasoning.
- While higher-resolution visual encoders generally correlate with improved performance, simply scaling up model size or increasing the number of input frames does not inherently resolve the underlying hallucination issues.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.