Reproduce “Align and Aggregate: Compositional Reasoning with Video Alignment and Answer Aggregation for Video Question-Answering” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhaohe Liao, Jiangtong Li, Li Niu, Liqing Zhang
VA³ augments existing VideoQA models with question-specific video alignment and graph-based answer aggregation, allowing them to select relevant clips and exchange information across decomposed sub-questions. Across AGQA-Decomp, novel-composition, longer-reasoning, MSVD, and NExT-QA settings, it improves both answer accuracy and compositional consistency, with gains of up to 4.76 points in accuracy and 5.03 points in consistency F1.
Key insights
- Question-specific temporal selection improves visual evidence for both main questions and sub-questions, while graph-based information sharing is needed to explicitly improve compositional consistency.
- The aligner and aggregator provide complementary benefits: the former focuses the model on relevant clips, whereas the latter links answers according to the structure of the reasoning graph.
- Correct final answers are not sufficient evidence of reliable reasoning; balanced parent-child consistency measures reveal whether improvements are supported by compatible sub-answer predictions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.