Reproduce “Visual Programmability: A Guide for Code-as-Thought in Chart Understanding” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Bohao Tang, Yan Ma, Fei Zhang, Jiadi Su, Ethan Chern, Zhulin Hu, Zhixin Wang, Pengfei Liu and 1 more

Researchers from Shanghai Jiao Tong University and Fudan University introduced "Visual Programmability," a learnable property that guides Vision-Language Models to adaptively select between code-based and direct visual reasoning for chart understanding. This framework achieved 62.8% average accuracy across four diverse benchmarks, outperforming fixed-strategy and state-of-the-art models by intelligently reducing code usage on low-programmability charts and enhancing data fidelity.

Key insights

  • Chart understanding requires task-dependent reasoning, with code-based approaches excelling on highly programmable charts and direct visual analysis for low-programmability cases.
  • Meta-cognitive skills, such as dynamically selecting the optimal reasoning strategy, are crucial for VLMs to achieve robust generalization across diverse chart complexities.
  • A comprehensive, multi-component reinforcement learning reward system, including specific incentives for correct strategy selection and data fidelity, is vital for training adaptive and reliable chart understanding models.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.