Reproduce “From Drawings to Decisions: A Hybrid Vision-Language Framework for Parsing 2D Engineering Drawings into Structured Manufacturing Knowledge” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Muhammad Tayyab Khan, Lequn Chen, Zane Yong, Jun Ming Tan, Wenhe Feng, Seung Ki Moon

The system extracts structured manufacturing fields from engineering drawings by detecting rotated annotation regions, cropping them, and parsing each patch into JSON. It is trained on 1,367 drawings across nine categories. On patch-level exact key-value evaluation, Donut reports 89.2% precision and 94.0% F1, versus Florence-2’s 78.4% and 85.0%. This comparison covers localized patches, not whole-drawing parsing or manufacturing outcomes.

Key insights

  • A two-stage approach, separating annotation localization from structured parsing, effectively addresses the inherent challenges of complex and densely annotated engineering drawings.
  • Fine-tuning open-source VLMs like Donut on a domain-specific dataset significantly improves accuracy, schema compliance, and reduces hallucination rates compared to general-purpose or zero-shot models, especially for symbol-heavy engineering content.
  • Curating a comprehensive, manually annotated dataset with oriented bounding boxes and structured JSON labels is crucial for training robust and accurate AI models tailored for engineering drawing interpretation.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.