Reproduce “MME-Industry: A Cross-Industry Multimodal Evaluation Benchmark” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Dongyi Yi, Guibo Zhu, Chenglin Ding, Zongshu Li, Dong Yi, Jinqiao Wang
Wuhan AI Research introduces MME-Industry, a new cross-industry multimodal evaluation benchmark designed to assess MLLMs' professional reasoning and decision-making capabilities across 21 distinct industrial sectors. Evaluations reveal that state-of-the-art MLLMs, particularly larger models, perform well in visually rich technical domains but show limitations in more abstract or nuanced industrial contexts.
Key insights
- All question-answer pairs were manually crafted and validated by domain experts from each industry, ensuring practical relevance, technical precision, and preventing data leakage.
- The benchmark deliberately removes OCR-based questions, focusing instead on assessing models' capabilities in complex industrial scenarios requiring deep visual understanding and domain-specific reasoning.
- The inclusion of an 'E' option for 'No corresponding features in the picture' or 'model unable to recognize/decode features' serves as a critical control for model robustness and instruction adherence.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.