Reproduce “Bridging Vision Language Models and Symbolic Grounding for Video Question Answering” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Haodi Ma, Vyom Pathak, Daisy Zhe Wang

University of Florida researchers introduced SG-VLM, a framework that integrates symbolic scene graph grounding with frozen Vision Language Models (VLMs) for Video Question Answering. This approach enhances VLM performance on specific VQA tasks by improving temporal and causal understanding through question-aware scene graph selection, demonstrating gains over prior modular reasoning systems.

Key insights

  • Question-aware scene graph selection is critical for VLM performance; irrelevant scene graphs introduce noise and degrade reasoning, with 'FrameSel-SG' consistently outperforming full graph integration.
  • Object-level symbolic grounding ('Summary-SG') is robust and often performs competitively, sometimes even better than including relations, suggesting reliable object detection provides more consistent value than potentially noisy relation extraction.
  • The benefits of symbolic grounding are not uniform but are most pronounced for specific question types and datasets, such as human-object interaction videos (iVQA) and object-centric or temporally specific questions.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.