Reproduce “MASR: Self-Reflective Reasoning through Multimodal Hierarchical Attention Focusing for Agent-based Video Understanding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shiwen Cao, Zhaoxing Zhang, Junming Jiao, Juyi Qiao, Guowen Song, Rong Shen, Xiangbing Meng
Researchers from Li Auto Inc. developed MASR, an agent-based framework that employs multimodal hierarchical attention and an efficient single-LLM self-reflection mechanism for video understanding. This system achieved state-of-the-art performance, with a 5% accuracy gain on the EgoSchema benchmark and strong results across other challenging long-form video QA datasets.
Key insights
- A hierarchical, multimodal approach to relevance sensing (MCRS) is effective for dynamically identifying and focusing on query-relevant segments in complex video data, avoiding inefficient global processing.
- Dilated Temporal Expansion (DTE) successfully broadens the temporal context around key frames, demonstrating that expanding the receptive field without resolution loss is vital for extracting comprehensive semantic information.
- Using a single Large Language Model for both response generation and self-reflection (confidence evaluation and context re-selection) offers an efficient and robust mechanism for iterative improvement in video understanding.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.