Reproduce “EgoMem: Lifelong Memory Agent for Full-duplex Omnimodal Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yiqun Yao, Naitong Yu, Xiang Li, Xin Jiang, Xuezhi Fang, Wenjia Ma, Xuying Meng, Jing Li and 2 more

EgoMem introduces a lifelong memory agent for full-duplex omnimodal models, enabling real-time user recognition, personalized responses, and long-term knowledge maintenance, including social relations, from raw audiovisual streams. The system achieves high accuracy in retrieval (face verification 98.4%, speaker recognition 96.5%) and dialog boundary detection (Jaccard score 0.992), leading to a Fact Score of 0.959 for personalized dialog in clean conditions.

Key insights

  • The asynchronous and modular design of EgoMem enables efficient processing of continuous, real-time omnimodal streams for lifelong memory management without introducing significant latency.
  • Integrating robust real-time user identification from raw audiovisual data, coupled with accurate dialog boundary detection, facilitates truly implicit and continuous personalization in dynamic interaction environments.
  • Leveraging external LLMs within the memory management process enhances robustness to ASR noise and formalizes knowledge extraction, allowing for effective continuous updates of structured memory and social relation graphs.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.