Reproduce “Scene-VLM: Multimodal Video Scene Segmentation via Vision-Language Models” on OpenResearch

About this paper

Nimrod Berman, Adam Botach, Emanuel Ben-Baruch, Shunit Haviv Hakimi, Asaf Gendler, Ilan Naiman, Erez Yosef, Igor Kviatkovsky

Scene-VLM, a fine-tuned Vision-Language Model framework developed by researchers from Ben-Gurion University, Amazon Prime Video, and Tel-Aviv University, advances video scene segmentation by integrating multimodal input and sequential prediction to achieve 62.1 F1 and 66.8 Average Precision on MovieNet-318, setting new state-of-the-art, and generating human-interpretable rationales for boundary decisions.

Key insights

• Fine-tuning a Vision-Language Model significantly improves video scene segmentation, outperforming zero-shot VLM applications and previous dedicated encoder-based methods.
• Multimodal input is crucial, with visual frames as the primary cue, but subtitles and character metadata provide valuable complementary signals, as indicated by attention analysis.
• Sequential prediction within a context-focus window enhances decision coherence and maintains consistent performance across various prediction positions, demonstrating the benefit of temporal dependencies.

Summary by alphaXiv.