Reproduce “OmniCAD: A Large-Scale Benchmark for 3D Spatial Reasoning in Robotics Assemblies” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Mingjia Wang, Taiting Lu, Ziwei Dong, Sisong Bei, Jingying Zeng, Runze Liu, Kaiyuan Lin, Hongxing Pan and 20 more
OmniCAD benchmarks models’ ability to identify industrial CAD assembly components, place them in six-degree-of-freedom poses, and recover mate graphs. It contains 25,000 assemblies with CAD annotations and rendered views. Across nine models, average exact inventory reconstruction is 95.1% without repeated parts and 26.4% when parts are reused, showing a hard evaluation task, not proven buildability.
Key insights
- Current state-of-the-art VLMs show limited proficiency in precise 6-DoF pose estimation and achieving physically valid, collision-free assemblies in industrial contexts.
- Identifying and correctly placing repeated component instances within an assembly is a significantly harder task for VLMs compared to non-repeated parts, indicating a challenge in multi-instance reasoning.
- While iterative visual feedback and geometric conflict reports provide some support for coarse error correction, fine-grained, collision-free alignment and accurate mate-graph recovery remain substantial challenges for current models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.