Reproduce “Vinci2: Providing Proactive Assistance in Continuous Egocentric Videos” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Gong Sitong, Tianyu Yan, Caixin Kang, Bo Zheng, Xiang Ruan, Huchuan Lu, Kaipeng Zhang, Yoichi Sato and 1 more
EgoMemo makes unsolicited assistance decisions from streaming first-person video by building timestamped captions, summaries, a knowledge graph, and visual memory as footage arrives. It retrieves semantic and visual context before choosing an intervention or silence. On EgoServe, it achieves macro-averaged F1 of 8.0, versus 4.7 for GPT-5-mini; the scores measure trigger and category matching, not user benefit or response quality.
Key insights
- Truly proactive assistance requires an explicit, context-dependent decision-making mechanism for *when* and *whether* to intervene, rather than just detecting events.
- Effective long-horizon temporal reasoning in egocentric videos necessitates hierarchical memory structures, semantic linking via a knowledge graph, and visual cues to synthesize comprehensive user context.
- A training-free, retrieval-augmented reasoning approach, combined with VLM-based caption reconstruction, can successfully enable proactive assistance in streaming video environments by integrating diverse contextual information.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.