Reproduce “MM-Ego: Towards Building Egocentric Multimodal LLMs for Video QA” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hanrong Ye, Haotian Zhang, Erik Daxberger, Lin Chen, Zongyu Lin, Yanghao Li, Bowen Zhang, Haoxuan You and 4 more
This paper presents MM-Ego, a framework for egocentric video understanding that introduces a large-scale egocentric QA dataset and a new benchmark, EgoMemoria, with a debiasing evaluation method. The core contribution is a multimodal LLM, MM-Ego, which utilizes "Memory Pointer Prompting" to efficiently process long egocentric videos, achieving an average Mean Debiased Accuracy of 61.27% on EgoMemoria and surpassing other general MLLMs including GPT-4o in this specialized domain.
Key insights
- Human-annotated narrations from existing datasets can be effectively leveraged by LLMs (like GPT-4o) to generate large-scale, high-quality multimodal QA data without relying on VLM-based labeling.
- Evaluating MLLMs for egocentric video understanding requires de-biasing methods to accurately assess true visual comprehension, as language-biased questions can inflate scores.
- A two-step "global glimpse" followed by "fallback" to high-resolution details (Memory Pointer Prompting) allows MLLMs to efficiently process and reason over very long videos while retaining critical fine-grained information.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.