Reproduce “Object-aware Video-language Pre-training for Retrieval” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Alex Jinpeng Wang, Yixiao Ge, Guanyu Cai, Rui Yan, Xudong Lin, Ying Shan, Xiaohu Qie, Mike Zheng Shou
The Object-aware Video-language Pre-training for Retrieval (OA-Trans) framework enhances video-text understanding by integrating fine-grained object-level information using a single masked anchor frame and a novel 4-stream contrastive loss. This method achieves state-of-the-art performance in video-text retrieval, demonstrating improved generalization without adding inference-time computational overhead.
Key insights
- Leveraging object information from just one anchor frame per video during pre-training is sufficient to achieve fine-grained semantic alignment without increasing inference costs or requiring object detection at test time.
- The multi-stream OAC loss effectively combines global video-text alignment with object-centric regularization, leading to more discriminative and semantically rich representations.
- The model learns a generalized object-centric understanding, capable of attending to and comprehending novel object categories not explicitly included in the pre-defined object tag dictionary.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.