Reproduce “Large Model based Sequential Keyframe Extraction for Video Summarization” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Kailong Tan, Yuxiang Zhou, Qianchen Xia, Rui Liu, Yong Chen
This paper introduces the Large Model based Sequential Keyframe Extraction (LMSKE) method for video summarization, which uses TransNetV2 for shot segmentation and CLIP for feature extraction, combined with an adaptive clustering algorithm and redundancy elimination. LMSKE achieved improved F1 scores and temporal coherence on the TVSum20 benchmark dataset while automatically determining the optimal number of keyframes.
Key insights
- Leveraging large pre-trained models like TransNetV2 for shot detection and CLIP for semantic feature extraction significantly enhances the quality and representativeness of video frames.
- An adaptive clustering algorithm, which automatically determines the optimal number of clusters based on the silhouette coefficient, improves robustness and eliminates the need for manual tuning.
- A sequential pipeline encompassing shot segmentation, adaptive clustering, and redundancy elimination ensures that extracted keyframes are representative, non-redundant, and temporally coherent.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.