Reproduce “VLog: Video-Language Models by Generative Retrieval of Narration Vocabulary” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Kevin Qinghong Lin, Mike Zheng Shou
VLog, a generative retrieval framework from Show Lab at the National University of Singapore, offers a new approach to video understanding by replacing traditional subword vocabularies with a "narration vocabulary" to generate real-time, event-driven video descriptions. This method achieves a 20x speedup over generative models, demonstrates improved reasoning capabilities on the Vidcab-Eval benchmark, and effectively adapts to new tasks including out-of-vocabulary event handling.
Key insights
- Replacing subword vocabularies with an event-driven "narration vocabulary" enables significantly faster and more concise video understanding, drawing inspiration from human cognitive processes for recalling experiences.
- A "generative retrieval" architecture can effectively integrate the complex reasoning abilities of generative models with the efficiency and flexible vocabulary management characteristic of retrieval models.
- Hierarchical indexing combined with an agentic vocabulary upgrading mechanism facilitates efficient navigation through large vocabularies and robust adaptation to novel events without requiring extensive model retraining.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.