Reproduce “Multi-Modal Answer Validation for Knowledge-Based VQA” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jialin Wu, Jiasen Lu, Ashish Sabharwal, Roozbeh Mottaghi

Multi-modal Answer Validation using External knowledge (MAVEx) addresses the challenge of noisy knowledge in open-domain Visual Question Answering by proposing an answer validation paradigm. The system achieves state-of-the-art results on the OK-VQA dataset by guiding knowledge retrieval with answer candidates and integrating external visual and textual information.

Key insights

  • Knowledge retrieval becomes significantly more relevant and helpful when guided by a potential answer candidate rather than just the question and image.
  • Integrating multi-modal external knowledge, specifically external visual knowledge from image search alongside textual and commonsense knowledge, complements traditional sources for richer evidence.
  • Aggregating retrieved knowledge at multiple granularities (query, noun-phrase, and question levels) through attention mechanisms is crucial for effectively utilizing diverse information.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.