Reproduce “From Recognition to Cognition: Visual Commonsense Reasoning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Rowan Zellers, Yonatan Bisk, Ali Farhadi, Yejin Choi

This research defines and addresses Visual Commonsense Reasoning, a new task where AI systems must answer questions about images and justify their responses with explicit rationales. A new dataset, VCR, featuring 290,000 examples from movie scenes, is introduced alongside an innovative Adversarial Matching technique to reduce dataset biases. The proposed R2C model achieves 44.0% accuracy on the holistic task, compared to 85.0% for humans.

Key insights

  • Adversarial Matching effectively minimizes dataset biases by ensuring plausible, yet incorrect, distractors, forcing models to rely on genuine multimodal reasoning rather than superficial patterns.
  • Strong, contextualized textual representations, such as those provided by BERT, are paramount for achieving robust performance in multimodal commonsense reasoning tasks.
  • A significant gap remains between human (85.0%) and state-of-the-art machine (44.0%) performance on Visual Commonsense Reasoning, indicating substantial room for future research in cognitive visual understanding.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.