Reproduce “Retrieval-Augmented Egocentric Video Captioning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jilan Xu, Yifei Huang, Junlin Hou, Guo Chen, Yuejie Zhang, Rui Feng, Weidi Xie
EgoInstructor is a retrieval-augmented framework that improves egocentric video captioning by drawing knowledge from third-person instructional videos. It achieves this by creating pseudo ego-exo video pairs from distinct datasets and utilizing a novel contrastive loss to align cross-view representations, leading to enhanced performance on various egocentric and cross-view understanding benchmarks.
Key insights
- Transferring knowledge from abundant third-person instructional videos can significantly enhance egocentric video understanding without requiring synchronously paired data.
- Refining exocentric video transcripts with large language models (LLMs) significantly improves visual-text alignment and cross-view retrieval performance.
- A specialized contrastive loss, EgoExoNCE, effectively bridges the domain gap by mining cross-view semantic relationships from language, promoting robust alignment across different perspectives and modalities.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.