Reproduce “CoVA-SFT: A Large-Scale Dataset for Chain of Visual Abstractions” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Tsung-Han Wu, Heekyung Lee, Anya Ji, Haoming Chen, Trevor Darrell, Joseph E. Gonzalez, David M. Chan

UC Berkeley researchers developed CoVA-SFT, a large-scale dataset for training multimodal language models to generate and integrate visual abstractions during complex reasoning, alongside CoVA-Bench for evaluation. Models fine-tuned on CoVA-SFT demonstrated improved performance in graph reasoning tasks by 62.0% compared to text-only baselines (57.0%), by leveraging internal visual workspaces.

Key insights

  • Internalizing visual workspaces, as opposed to external tool calls, mitigates error propagation issues in multi-step interleaved reasoning, leading to more coherent and robust reasoning trajectories.
  • The utility of latent visual abstractions is task-dependent; they offer clear advantages for structurally motivated tasks like graph reasoning, which are cumbersome to represent textually.
  • For tasks already efficiently handled by symbolic textual notation (e.g., certain mathematical reasoning), latent visual tokens can be an "expensive detour" rather than a beneficial aid, indicating domain-specific optimality.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.