Reproduce “Masked Video Distillation: Rethinking Masked Feature Modeling for Self-supervised Video Representation Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Rui Wang, Dongdong Chen, Zuxuan Wu, Yinpeng Chen, Xiyang Dai, Mengchen Liu, Lu Yuan, Yu-Gang Jiang
Masked Video Distillation (MVD) introduces a two-stage self-supervised pretraining framework for video transformers. It distills high-level features from pre-trained masked image modeling (MIM) and masked video modeling (MVM) teachers, overcoming limitations of low-level pixel reconstruction by employing a spatial-temporal co-teaching strategy. MVD achieves new state-of-the-art results on Kinetics-400 (87.2%) and Something-Something V2 (77.3%), demonstrating improved efficiency and superior performance across various video benchmarks.
Key insights
- Distilling high-level features from pre-trained teachers provides a more robust and semantically richer signal for self-supervised video representation learning than reconstructing noisy low-level pixels.
- Image-trained models excel at providing spatial representations, while video-trained models are better at capturing spatial-temporal dynamics, confirming their complementary strengths for video understanding.
- Combining knowledge from both image and video teachers through a co-teaching strategy results in a synergistic effect, leading to more balanced and powerful video representations that perform well across diverse tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.