Reproduce “CLaMR: Contextualized Late-Interaction for Multimodal Content Retrieval” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
David Wan, Han Wang, Elias Stengel-Eskin, Jaemin Cho, Mohit Bansal
A system developed at UNC Chapel Hill, CLAMR, improves video retrieval by dynamically selecting the most relevant modality from video content using joint encoding and contextualized late-interaction. It achieves a 25.6 point increase in nDCG@10 on MULTIVENT 2.0++ over single-modality baselines and 5.9 percentage points R@1 over single-modality baselines on MSR-VTT.
Key insights
- Joint encoding of all modalities (video frames, ASR, OCR, metadata) is crucial for learning contextualized relationships and significantly improves retrieval performance.
- Traditional fixed aggregation methods for multimodal content often degrade performance compared to using the best single modality, underscoring the necessity of dynamic modality selection.
- Modality-aware training, combined with synthetically generated modality-targeted queries, enables the model to effectively learn and select the most relevant modality for a given query.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.