Reproduce “Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptions” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hongwei Xue, Tiankai Hang, Yanhong Zeng, Yuchong Sun, Bei Liu, Huan Yang, Jianlong Fu, Baining Guo
Microsoft Research Asia developed HD-VILA-100M, a comprehensive dataset featuring 100 million high-resolution (720p) video clip-sentence pairs, and proposed the HD-VILA model, which employs an efficient hybrid video encoder to achieve state-of-the-art performance in video-language understanding and text-to-visual generation.
Key insights
- High-resolution video content and semantic diversity in large-scale datasets are critical for superior cross-modality learning and model generalization.
- A hybrid video encoder effectively balances computational efficiency with detailed spatiotemporal feature learning from high-resolution video.
- Joint pre-training with VLM and MLM, even on noisy ASR data, successfully aligns visual and linguistic representations for robust performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.