Reproduce “DeepSketcher: Internalizing Visual Manipulation for Multimodal Reasoning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Chi Zhang, Haibo Qiu, Qiming Zhang, Zhixiong Zeng, Lin Ma, Jing Zhang
DeepSketcher enables Vision Language Models (VLMs) to internalize visual manipulation, fostering deeper multimodal reasoning by dynamically interacting with visual inputs. The model, trained on a meticulously code-rendered dataset, achieved an average accuracy of 46.0% across five benchmarks, improving the base model's performance by 3.9 percentage points and outperforming other "inner visual thought" and "tool-calling" VLMs.
Key insights
- Code-rendered images and code-space manipulations provide a robust, precise, and reproducible foundation for training VLMs in complex visual reasoning and manipulation tasks, mitigating grounding noise.
- Internalizing visual manipulation through an embedding editor in the latent space allows for a more flexible, efficient, and self-contained "thinking with images" paradigm compared to external tool-calling mechanisms.
- A multi-stage, decoupled training process is crucial for effectively integrating visual manipulation capabilities with an LLM's reasoning skills, ensuring stability and better overall performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.