Reproduce “LifelongMemory: Leveraging LLMs for Answering Queries in Long-form Egocentric Videos” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ying Wang, Yanlai Yang, Mengye Ren
Researchers from NYU's Agentic Learning AI Lab introduced LifelongMemory, a framework that processes long-form egocentric videos for natural language querying by converting video into textual digests for LLM reasoning. It achieved state-of-the-art accuracy on EgoSchema QA (64.7%) and competitive performance on Ego4D NLQ, demonstrating robust understanding of personal video memories.
Key insights
- The Caption Digest module significantly improves efficiency and relevance for LLM reasoning by creating a concise textual representation of long videos.
- Explicitly prompting LLMs for explanations and confidence scores enhances their reasoning process, leading to improved performance and calibrated confidence levels.
- Employing egocentric-specific MLLMs for captioning and powerful LLMs for reasoning is crucial, with GPT-4/GPT-4o outperforming weaker models in both accuracy and instruction following.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.