Reproduce “IR3D-Bench: Evaluating Vision-Language Model Scene Understanding as Agentic Inverse Rendering” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Parker Liu, Chenxin Li, Zhengxin Li, Yipeng Wu, Wuyang Li, Zhiqin Yang, Zhenyuan Zhang, Yunlong Lin and 2 more

IR3D-Bench tests whether vision-language models can reconstruct scenes from images by returning JSON object attributes and positions that a pipeline renders in Blender. On 1,500 CLEVR image-scene pairs, Gemini-2.5-Pro reaches 0.55 pairwise relation accuracy and 93.08 overall appearance CLIP, while GPT-4o scores 0.30 and 94.22. These results indicate appearance matching need not ensure relational accuracy in this controlled synthetic setting.

Key insights

  • Models tend to recognize object categories and appearance more reliably than they recover precise geometry, overlap, or spatial layout.
  • Performance depends on the metric: different systems lead on localization, relations, counts, or appearance, so no single score captures reconstruction quality.
  • A structured output format is essential for rendering, while iterative comparison with a rendered prediction can help models revise their scene estimates.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.