Reproduce “Perceive, Query & Reason: Enhancing Video QA with Question-Guided Temporal Queries” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Roberto Amoroso, Gengyuan Zhang, Rajat Koner, Lorenzo Baraldi, Rita Cucchiara, Volker Tresp
A new framework, PQR, enhances Video Question Answering by integrating a T-Former, a question-guided temporal querying Transformer, which efficiently extracts contextually relevant visual features. This approach improves performance on several video QA benchmarks, achieving a 2.9% accuracy gain on NExT-QA and a 2.2% gain on STAR, using a smaller LLM and without extensive video pre-training.
Key insights
- T-Former effectively addresses spatio-temporal information extraction in videos by providing question-guided, contextually relevant input to LLMs, mitigating computational complexity.
- Dynamic, clustering-based sampling methods (e.g., k-medoids) for initializing temporal queries in T-Former significantly outperform static or random methods, yielding more effective results.
- The PQR framework achieves competitive performance on video QA benchmarks without requiring extensive video-based pre-training, differentiating it from many other MLLM-based video QA systems.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.