Reproduce “InSituMeasure: Probing Situated Measurement Grounding in Industrial Scenes with Multimodal Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Chao Shen, Xinyuan Li, Yunfan Zhou, Jianguo Yao, Haibing Guan, Zhihai Wang, Xijun Li
The authors introduce InSituMeasure, a benchmark for evaluating multimodal large language models on gauge reading in authentic industrial scenes. It tests whether systems select the right instrument, interpret its value and unit, and abstain when evidence is insufficient. Across 24 models, the best joint value-and-unit accuracy is 25.7% on answerable tasks, showing benchmark difficulty, not field performance.
Key insights
- A significant performance gap exists between MLLMs' general multimodal capabilities and their reliability in situated, continuous-valued measurement tasks within complex industrial environments.
- MLLM failures often stem from a combination of factors, termed 'Mixed Causes' (e.g., viewpoint deviation, occlusion, environmental interference), rather than isolated visual degradations, impairing localization, scale interpretation, and unit grounding.
- MLLMs exhibit poor confidence calibration, frequently providing overconfident and incorrect predictions on ambiguous or unanswerable tasks, which could pose operational or safety risks in industrial settings.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.