Reproduce “Grounded Question-Answering in Long Egocentric Videos” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shangzhe Di, Weidi Xie
Researchers at Shanghai Jiao Tong University developed GroundVQA, a unified model capable of simultaneously performing temporal grounding and open-ended question-answering in long egocentric videos. The work also introduces EGOTIMEQA, a 30-fold larger dataset generated using Large Language Models, which significantly enhanced model performance and addressed data scarcity, enabling GroundVQA to achieve state-of-the-art results on the QAEGO4D benchmark.
Key insights
- Unified training for temporal grounding and question-answering in long videos significantly reduces error propagation and improves understanding of query-relevant information.
- Large Language Models can automatically synthesize extensive, high-quality question-answering datasets from existing narrations, effectively addressing data scarcity in egocentric video tasks.
- Integrating a close-ended QA task and using sentence similarity metrics offers a more comprehensive and reliable approach for evaluating open-ended answer generation in multi-modal systems.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.