Reproduce “From Representation to Reasoning: Towards both Evidence and Commonsense Reasoning for Video Question-Answering” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jiangtong Li, Li Niu, Liqing Zhang
Researchers at Shanghai Jiao Tong University developed Causal-VidQA, a large-scale video question-answering dataset with 26,900 video clips and 107,600 question-answer pairs, designed to challenge models with both evidence and commonsense reasoning tasks. The dataset uniquely requires explicit justifications for commonsense inferences, revealing a substantial performance gap where state-of-the-art models like B2A with BERT achieve 35.16% accuracy for counterfactual reasoning compared to human performance of 92-95%.
Key insights
- Diagnostic experiments confirmed that the Causal-VidQA dataset is free from significant statistical biases, compelling models to engage in genuine video and language understanding rather than relying on shortcuts.
- Advanced language models like BERT consistently and substantially improved performance across all reasoning types compared to simpler word embeddings, highlighting their capability in integrating commonsense knowledge for inferential tasks.
- Graph-based VideoQA models generally performed better than other architectures, suggesting that modeling complex inter- and intra-modality relationships is crucial for addressing deeper video reasoning challenges.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.