Reproduce “FOLIO: Focused Semantic Memory for Streaming Video Understanding” on OpenResearch

About this paper

Haoyang Fan, Dhruv Parikh, Anvitha Ramachandran, Sameh Gobriel, Nilesh Jain, Rajgopal Kannan, Viktor Prasanna

FOLIO, developed by researchers from the University of Southern California and Intel Labs, introduces a focused, entity-centered semantic memory system for online streaming video understanding. It achieves state-of-the-art accuracy on single-turn (OVO-Bench) and multi-turn (StreamingBench) benchmarks, with an average 6.6% accuracy improvement while reducing memory writing costs by over 30% through dynamic focus and adaptive writing.

Key insights

• Organizing long-term memory around stable entity identities, rather than raw visual data or temporal events, significantly improves retrieval for target-specific queries and multi-turn reference resolution.
• A dynamic focus state effectively manages memory cost and redundancy by selectively recording critical information with higher detail, leading to substantial reductions in token and record generation.
• The short-term visual buffer is indispensable for grounding current-state queries, while the long-term semantic memory provides essential historical recall, demonstrating the value of a hybrid memory approach.

Summary by alphaXiv.