Reproduce “EgoAVU: Egocentric Audio-Visual Understanding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ashish Seth, Xinhao Mei, Changsheng Zhao, Varun Nagaraja, Ernie Chang, Gregory P. Meyer, Gael Le Lan, Yunyang Xiong and 4 more
Researchers from Meta and the University of Maryland, College Park developed EgoAVU, an automated data engine that creates large-scale egocentric audio-visual-language datasets. This system trains multimodal large language models to overcome vision bias and substantially improves their ability to understand complex first-person audio-visual cues, showing performance gains up to 113.3% on specific reasoning tasks.
Key insights
- Current MLLMs exhibit a strong 'vision bias,' frequently neglecting audio cues or misgrounding them in egocentric video understanding tasks.
- Modular generation of unimodal narrations combined with a structured Multi-modal Context Graph (MCG) is effective in creating coherent joint audio-visual descriptions, circumventing direct MLLM-based merging issues.
- MLLMs possess a self-learning potential, where robust unimodal capabilities can be leveraged to generate data that significantly enhances their joint audio-visual reasoning and transferability.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.