Reproduce “Knowledge Regularized Negative Feature Tuning of Vision-Language Models for Out-of-Distribution Detection” on OpenResearch

About this paper

Wenjie Zhu, Yabin Zhang, Xin Jin, Wenjun Zeng, Lei Zhang

Knowledge Regularized Negative Feature Tuning (KR-NFT) enhances out-of-distribution (OOD) detection in Vision-Language Models (VLMs) by efficiently adapting text features while preserving pre-trained knowledge. The method achieves improved generalization, demonstrating an average AUROC of 89.25% on unseen OOD classes and 92.88% on unseen styles, outperforming prior state-of-the-art methods.

Key insights

• Directly tuning pre-trained text features with a lightweight, image-conditional module can efficiently adapt VLMs for OOD detection while maintaining fast inference.
• Explicitly regularizing tuned features to preserve their proximity to original pre-trained features is crucial for mitigating catastrophic forgetting and enhancing generalization to unseen data.
• Employing separate, distribution-aware transformations for positive and negative text features significantly improves the model's ability to discriminate between in-distribution and out-of-distribution samples.

Summary by alphaXiv.