Reproduce “Predict, Then Retrieve: Cross-Instance Future-State Retrieval from Video Prefixes” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Quynh Vo, Thong Nguyen, Vinh-Hien Do, Cong-Duy Nguyen, Anh-Tuan Luu
The Predictive State Retrieval (PSR) benchmark is introduced, defining a new task that requires models to anticipate an object's future state from a video prefix and retrieve cross-instance visual examples of that state. The LFTR baseline achieved an R@5 of 16.1 1.0, outperforming a 32B multimodal language model and demonstrating that forecasting, rather than perception, is the primary challenge.
Key insights
- Cross-space fusion of semantic and visual retrieval spaces is critical, providing the largest performance gains by leveraging complementary information about future states.
- Hard negative training is essential for improving visual space retrieval, specifically in mitigating visual confusability among potential future states.
- Forecasting future states, rather than perception or retrieval mechanisms, represents the central learnable bottleneck, particularly for predictions beyond short temporal horizons.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.