Reproduce “DINO-QPM: Adapting Visual Foundation Models for Globally Interpretable Image Classification” on OpenResearch

About this paper

Robert Zimmermann, Thomas Norrenbrock, Bodo Rosenhahn

DINO-QPM introduces a lightweight adapter that transforms the feature representations of frozen DINOv2 visual foundation models into contrastive, class-independent, and human-interpretable features for image classification. It achieves 88.3% accuracy on CUB-2011 and 94.0% on Stanford Cars, outperforming linear probes while offering globally interpretable and spatially localized explanations with 95.0% plausibility on CUB-2011.

Key insights

• Interpretable-by-design methods can be effectively adapted to powerful, frozen visual foundation models, providing a pathway to high-performance and transparent AI systems without extensive computational overhead.
• Directly leveraging average-pooled patch embeddings from a VFM, rather than global CLS tokens, is crucial for achieving high-fidelity spatial localization of interpretable features.
• Introducing specific L1 sparsity losses, particularly on feature maps, significantly enhances explanation 'Plausibility' by concentrating feature activations on semantically relevant object parts and suppressing background noise.
• The work demonstrates that achieving strong interpretability does not necessitate a trade-off with classification accuracy, as DINO-QPM surpasses uninterpretable linear probes in performance.

Summary by alphaXiv.