Reproduce “Multimodal Industrial Anomaly Detection via Hybrid Fusion” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yue Wang, Jinlong Peng, Jiangning Zhang, Ran Yi, Yabiao Wang, Chengjie Wang

Researchers from Shanghai Jiao Tong University and Tencent Youtu Lab developed Multi-3D-Memory (M3DM), a hybrid fusion scheme for multimodal industrial anomaly detection leveraging both 2D RGB images and 3D point clouds. The method achieves state-of-the-art results on the MVTec-3D AD dataset, with a 0.945 Image-level AUROC for detection and 0.964 AUPRO for segmentation.

Key insights

  • A hybrid fusion approach, combining inter-modal interaction (Unsupervised Feature Fusion) with multi-faceted decision-making (Decision Layer Fusion), improves upon simple feature concatenation for multimodal anomaly detection.
  • Spatially aligning irregular 3D point cloud features to a 2D grid (Point Feature Alignment) enables effective integration with 2D image features and leveraging powerful 2D-centric models.
  • Maintaining separate memory banks for individual modalities alongside fused features prevents information loss and leads to more robust anomaly detection, as some anomalies may only be apparent in a single modality.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.