Reproduce “Mug-STAN: Adapting Image-Language Pretrained Models for General Video Understanding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ruyang Liu, Jingjia Huang, Wei Gao, Thomas H. Li, Ge Li
Mug-STAN is a framework that adapts image-language pretrained models for general video understanding by incorporating a Spatial-Temporal Auxiliary Network (STAN) for robust temporal modeling and a Mutual-guided (Mug) alignment module to suppress partial video-text misalignment. This approach establishes new state-of-the-art results across various video tasks and enables zero-shot video chatting without specific instruction tuning.
Key insights
- Achieving effective image-to-video transfer requires addressing both robust temporal modeling and the pervasive issue of partial misalignment in video-text data simultaneously.
- A non-intrusive, branched auxiliary network that leverages multi-level features and reuses pretrained spatial parameters can effectively learn temporal dynamics without disrupting the core semantic knowledge of the original ILPM.
- A parameter-free mutual guidance mechanism can enhance cross-modal alignment by dynamically identifying and suppressing irrelevant information from both video and text, making post-pretraining on noisy datasets significantly more effective.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.