Reproduce “Fine-Tuning Vision-Language Model for Automated Engineering Drawing Information Extraction” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Muhammad Tayyab Khan, Lequn Chen, Ye Han Ng, Wenhe Feng, Nicholas Yew Jin Tan, Seung Ki Moon
The authors adapt Florence-2 to extract structured GD&T records from engineering drawings by fine-tuning it on expert-labeled examples. Exp-3 pairs the model with query augmentation and GPT-4o-mini formatting post-processing. It achieves 61.51% F1, a 52.40% relative increase over Claude-3.5-Sonnet’s 40.36% F1, but this comparison does not isolate fine-tuning and is limited to the tested setup and dataset.
Key insights
- Domain-specific fine-tuning of a smaller, open-source VLM can substantially outperform larger, general-purpose VLMs operating in a zero-shot setting for highly specialized industrial tasks like GD&T extraction.
- Increasing data augmentation through diverse query questions per image significantly boosts both the accuracy and recall capabilities of the fine-tuned model in capturing complex GD&T features.
- This approach provides a computationally efficient and cost-effective method for deploying advanced AI in manufacturing automation, reducing reliance on expensive proprietary APIs and making specialized solutions more accessible.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.