Reproduce “VTFusion: A Vision-Text Multimodal Fusion Network for Few-Shot Anomaly Detection” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yuxin Jiang, Yunkang Cao, Yuqi Cheng, Yiheng Zhang, Weiming Shen

VTFusion proposes a vision-text multimodal fusion network for few-shot anomaly detection, employing adaptive feature extractors and a multimodal prediction fusion module to bridge the domain gap between pre-trained models and industrial data. This approach consistently outperforms state-of-the-art methods on public and industrial datasets, yielding higher AUROC for both image-level detection and pixel-level localization of anomalies.

Key insights

  • Successfully adapting pre-trained Vision-Language Model (VLM) features to specific industrial domains is crucial for effective anomaly detection, overcoming the "domain gap" challenge.
  • Diversified synthetic anomaly generation effectively addresses data scarcity, allowing models to learn robust normal and abnormal feature boundaries with limited real samples.
  • Sophisticated multimodal fusion, employing mechanisms like self-attention and residual blocks, is essential for robustly integrating visual and textual predictions by mitigating semantic misalignment and cross-modal interference.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.