Reproduce “Visual Semantic Role Labeling for Video Understanding” on OpenResearch

About this paper

Arka Sadhu, Tanmay Gupta, Mark Yatskar, Ram Nevatia, Aniruddha Kembhavi

The paper introduces Visual Semantic Role Labeling for Videos (VidSRL), a new task that aims for a structured understanding of video content, along with VidSitu, a large-scale dataset of movie clips. This work enables models to predict verbs, semantic roles, entity co-references, and inter-event relations within 10-second video segments, providing a more detailed and interpretable representation of dynamic scenes.

Key insights

• VidSitu offers a uniquely diverse and complex benchmark for video understanding, featuring a wide range of verbs and entities, and significantly richer event descriptions (avg. 4.2 unique verbs, 6.5 unique entities, and 90% of events with 4+ roles per 10-second clip) compared to existing datasets.
• All VidSRL sub-tasks, particularly argument prediction, show that visual information is critical, with video-conditioned models significantly outperforming video-blind baselines.
• Current state-of-the-art video models, even with large-scale pre-training, still exhibit substantial gaps relative to human performance in VidSRL, especially for complex reasoning tasks like entity co-referencing and event relation prediction, indicating significant room for future research.

Summary by alphaXiv.