Reproduce “R4DSG: Relative 4D Scene Graph Memory for Object-Centric Question Answering in Long Egocentric Video” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ke Ma, Yamin Mao, Weiming Li, Shuai Tan, Yijie Zhong, Hao Chen, Haofen Wang, Meng Wang
R4DSG targets long first-person video memory by tracking persistent objects relative to stable landmarks and recording changes in their relations over time. Its time-indexed relative 3D scene-graph memory turns observations into searchable documents without requiring a global map. On the public EgoLifeQA A1_JAKE single-subject split, it reached 43.1% accuracy on the 72-question when subset, versus 30.6% for EgoRAG-Text.
Key insights
- Meaningful object-centric reasoning in egocentric video can be achieved by focusing on anchor-relative state transitions between static environmental anchors and dynamic objects, rather than requiring a globally consistent 3D world model.
- Maintaining persistent object identity across long temporal spans, coupled with tracking relative spatial relations to stable anchors, is crucial for addressing "where" and "when" questions effectively.
- A compact, intermediate-granularity memory representation that summarizes segment-level information while preserving critical relational and temporal cues is optimal for efficient retrieval-augmented question answering in long egocentric videos.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.