Reproduce “OmniVL:One Foundation Model for Image-Language and Video-Language Tasks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Junke Wang, Dongdong Chen, Zuxuan Wu, Chong Luo, Luowei Zhou, Yucheng Zhao, Yujia Xie, Ce Liu and 2 more
OmniVL unifies image-language and video-language understanding into a single foundation model, demonstrating that joint pretraining leads to mutual performance benefits across diverse image-only, video-only, cross-modal, and multi-modal tasks. The model sets new performance benchmarks on various downstream datasets, including achieving 86.7% average accuracy on image classification and 47.8 R@1 on MSRVTT text-to-video retrieval.
Key insights
- Unified pretraining enables bidirectional performance improvements, where image-language and video-language tasks mutually benefit from the joint learning paradigm.
- Integrating diverse data types, including web-crawled image/video-text pairs and human-annotated image/video-label data, yields more discriminative visual representations for both general and specific tasks.
- A single, unified architecture can inherently support a broad spectrum of functionalities—visual-only, cross-modal alignment, and multi-modal generation—without requiring task-specific adaptors.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.