Reproduce “OmniMech: All-in-one Multimodal Mechanical Benchmark for 3D Reconstruction” on OpenResearch

About this paper

Taiting Lu, Runze Liu, Ziwei Dong, Sisong Bei, Jingying Zeng, Mingjia Wang, Zhenghao Li, Kaiyuan Lin and 13 more

OmniMech benchmarks the translation of dimensioned mechanical drawings into editable 3D CAD models by pairing real-world drawings with native SolidWorks parts and geometric and visual representations. It evaluates CAD generation, geometry recovery, constraint interpretation, and tool use. In a 5,000-sample zero-shot test, GPT-5.5 achieved 92.6% pass rate but 41.6% volume error, so validity does not establish accuracy or manufacturing readiness.

Key insights

• Current state-of-the-art VLMs and specialized CAD models demonstrate substantial limitations, indicating a fundamental gap between visual resemblance and the high geometric precision and engineering fidelity required for industrial CAD.
• Providing richer visual information (e.g., multiple renders or video) generally improves geometric fidelity metrics, but these improvements are not consistently monotonic and do not resolve core issues in dimensional accuracy or feature interpretation.
• Agentic iterative refinement primarily enhances the reliability (pass rate) of CAD program execution and error correction, rather than fundamentally improving the model's intrinsic geometric accuracy or deep reasoning capabilities.

Summary by alphaXiv.