Reproduce “Thinking with Visual Primitives” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ruijie Lu, Yiyang Ma, Xiaokang Chen, Lingxiao Luo, Zhiyu Wu, Zizheng Pan, Xingchao Liu, Yutong Lin and 9 more
The authors introduce visual primitives into multimodal reasoning, allowing a model to emit bounding boxes and points as it reasons about images. Boxes identify regions, while points mark locations and path waypoints. On a 2,000-instance maze benchmark, their model reports 66.9% ACC, versus GPT-5.4’s 50.6%; this comparison does not isolate the effect of primitives or establish route validity.
Key insights
- The "Reference Gap" – the ambiguity of natural language in precisely referencing visual entities – is a fundamental limitation for MLLMs in complex reasoning, distinct from the "Perception Gap."
- Integrating visual primitives (points and bounding boxes) directly into an MLLM's thought process allows the model to "point" while it "reasons," anchoring abstract linguistic thoughts to concrete spatial coordinates and mitigating logical errors.
- An optimized architecture can achieve high visual token efficiency, processing complex images with a significantly reduced Key-Value (KV) cache budget (e.g., ~90 entries for an 800x800 image) while maintaining competitive reasoning performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.