Reproduce “Customizing Visual-Language Foundation Models for Multi-modal Anomaly Detection and Reasoning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xiaohao Xu, Yunkang Cao, Huaxin Zhang, Nong Sang, Xiaonan Huang
Researchers at the University of Michigan-Ann Arbor and Huazhong University of Science and Technology customized visual-language foundation models for multi-modal anomaly detection and reasoning by employing a unified pre-processing step and a multi-modal prompting strategy. This approach enhanced detection accuracy and enabled interpretable anomaly explanations across various industrial data, achieving an average accuracy of 84% with GPT4-V on the MVTec-AD dataset and competitive performance from open-source models like Qwen2VL-7B.
Key insights
- Sophisticated multi-modal prompting significantly enhances the accuracy and interpretability of anomaly detection by effectively integrating domain-specific knowledge into VLMs without requiring extensive retraining.
- Generic visual-language models can be adapted to perform anomaly detection across diverse data modalities by standardizing varied inputs into a 2D image format, broadening their applicability in industrial settings.
- Open-source visual-language models demonstrate competitive, and sometimes superior, performance compared to leading commercial models in specific anomaly detection tasks, showcasing the rapid advancement and potential of open-source AI.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.