Reproduce “EgoCross: Benchmarking Multimodal Large Language Models for Cross-Domain Egocentric Video Question Answering” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yanjun Li, Yuqian Fu, Tianwen Qian, Qi'ao Xu, Silong Dai, Danda Pani Paudel, Luc Van Gool, Xiaoling Wang
EgoCross introduces a benchmark for evaluating Multimodal Large Language Models (MLLMs) on cross-domain egocentric video question answering, revealing that current MLLMs struggle to generalize to specialized, unfamiliar contexts like surgery and extreme sports, with most scoring below 55% accuracy. The benchmark incorporates four distinct professional domains and demonstrates a significant performance drop for MLLMs compared to daily-life egocentric tasks.
Key insights
- Current state-of-the-art MLLMs, including those specialized for egocentric data, exhibit limited cross-domain generalization, performing poorly on specialized tasks outside common daily activities.
- The difficulty of cross-domain EgocentricQA varies significantly across domains, with Industry and Extreme Sports proving more challenging than Animal Perspective, correlating with their distinct visual and textual feature spaces.
- Reinforcement Learning (RL) approaches demonstrated the most promising improvements for cross-domain adaptation, suggesting that dynamic learning from interaction and feedback is crucial for tackling complex, out-of-distribution egocentric tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.