Reproduce “From Captions to Keyframes: KeyScore for Multimodal Frame Scoring and Video-Language Understanding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shih-Yao Lin, Sibendu Paul, Caren Chen
Amazon Prime Video researchers introduced KeyScore, a caption-aware frame scoring framework, and STACFP, a spatio-temporal clustering method, to efficiently select informative keyframes from long videos. This dual approach achieved a 99% frame reduction rate, while improving video-text retrieval R@1 scores by several percentage points and boosting zero-shot action classification F1 by 12.0 points on HMDB-51.
Key insights
- Intelligently integrating temporal information into frame clustering allows for more adaptive and efficient candidate frame proposal than uniform sampling or purely visual clustering.
- A multi-faceted scoring approach, combining semantic alignment and the unique contextual drop impact, effectively identifies keyframes that are both relevant to captions and indispensable to overall video understanding.
- Significant frame reduction (up to 99%) can be achieved simultaneously with improved, rather than degraded, performance in various video-language tasks when frames are selected based on semantic and contextual importance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.