Reproduce “FORGE: Fine-grained Multimodal Evaluation for Manufacturing Scenarios” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xiangru Jian, Hao Xu, Wei Pang, Xinjian Zhao, Chengyu Tao, Qixin Zhang, Xikun Zhang, Chao Zhang and 8 more
FORGE, a new multimodal benchmark and dataset, provides real-world 2D images and 3D point clouds with fine-grained manufacturing semantics to rigorously evaluate Multimodal Large Language Models (MLLMs) in industrial scenarios. The evaluation of 18 MLLMs reveals significant performance gaps and identifies domain knowledge as a key bottleneck, while also demonstrating that fine-tuning compact models on FORGE data can yield substantial accuracy improvements on unseen tasks.
Key insights
- Insufficient domain-specific manufacturing knowledge and the inability to perform fine-grained morphological analysis, rather than basic visual grounding, constitute the primary bottlenecks for MLLMs in manufacturing scenarios.
- For 3D data, multi-view image rendering (3V) is a more effective interface for general-purpose MLLMs than directly feeding raw point cloud coordinates as text tokens.
- Contextual examples through In-Context Demonstration (ICD) are more effective than simple visual references in improving MLLM performance by providing essential task logic and reasoning paths.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.