Reproduce “CRoFT: Robust Fine-Tuning with Concurrent Optimization for OOD Generalization and Open-Set OOD Detection” on OpenResearch

About this paper

Lin Zhu, Yifeng Yang, Qinying Gu, Xinbing Wang, Chenghu Zhou, Nanyang Ye

Researchers from Shanghai Jiao Tong University introduced CRoFT, a fine-tuning framework for Vision-Language Pre-trained Models that concurrently optimizes for out-of-distribution generalization and open-set OOD detection. This approach, which theoretically links OOD detection to generalization via Hessian matrices, achieved improved performance in both areas, including a 25.3% reduction in FPR95 for OOD detection in one setting and a 4.3% boost in OOD generalization accuracy.

Key insights

• A theoretical connection exists between open-set Out-of-Distribution (OOD) detection and OOD generalization, demonstrating that optimizing for one can benefit the other through domain-consistent Hessians.
• Minimizing the gradient magnitude of energy scores on in-distribution data effectively reshapes energy distributions, improving the discriminability of open-set OOD data.
• Adversarial generation of worst-case covariate-shifted features during training helps the model learn more robust adapters for improved generalization to various data styles.

Summary by alphaXiv.