Reproduce “Enhancing Long Video Understanding via Hierarchical Event-Based Memory” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Dingxin Cheng, Mingda Li, Jingyu Liu, Yongxin Guo, Bin Jiang, Qingbin Liu, Xi Chen, Bo Zhao
HEM-LLM, developed by researchers including those from Tencent PCG and Shandong University, enhances Large Language Models' ability to comprehend long videos by introducing a hierarchical event-based memory mechanism. This approach segments videos into distinct events and models both intra-event and inter-event temporal dependencies, achieving state-of-the-art results across various video understanding tasks, including a 19.4% improvement on the MovieChat-1K dataset while using significantly fewer frames.
Key insights
- Adaptive segmentation of long videos into events significantly reduces information redundancy and prevents the blending of unrelated semantics, which improves overall comprehension.
- A hierarchical memory architecture, comprising both local (intra-event) and global (inter-event) memories, enables comprehensive multi-granular temporal modeling crucial for understanding complex video narratives.
- Integrating a compressed global memory into the Q-Former's self-attention mechanism allows LLMs to effectively leverage historical event context for current event processing, enhancing reasoning over long durations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.