Reproduce “Synthesizing Visual Concepts as Vision-Language Programs” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Antonia Wüst, Wolfgang Stammer, Hikaru Shindo, Lukas Helff, Devendra Singh Dhami, Kristian Kersting

The Vision-Language Programs (VLP) framework integrates the flexible perception of pre-trained Vision-Language Models (VLMs) with systematic symbolic program synthesis to inductively learn visual concepts. VLP enables more interpretable, logically consistent visual reasoning by generating executable programs that explain visual rules, achieving performance improvements of up to 13.5% over direct VLM prompting on complex reasoning tasks while facilitating shortcut mitigation.

Key insights

  • Decoupling VLM perception from systematic symbolic reasoning creates a robust and generalizable neuro-symbolic architecture for inductive visual concept learning.
  • The generation of human-interpretable programs provides transparency into the reasoning process, allowing for debugging, understanding, and mitigation of shortcut learning.
  • VLP's approach offers a more computationally efficient and consistent method for complex visual reasoning compared to internal 'thinking' processes of large language models.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.