Reproduce “Contrastive Adapters for Foundation Model Group Robustness” on OpenResearch

About this paper

Michael Zhang, Christopher Ré

Stanford University researchers developed "contrastive adapters," an efficient method to enhance the group robustness of frozen foundation models without full finetuning. This approach consistently improved worst-group accuracy across nine benchmarks by up to 56.0 percentage points, matching or exceeding the performance of methods requiring full model retraining while using less than 1% of parameters.

Key insights

• While zero-shot classification with Foundation Models can fail on specific groups, the underlying pretrained embeddings often contain sufficient information for correct group-robust classification.
• Standard adapter training methods are insufficient for group robustness because they do not explicitly address the problem of "far apart" embeddings for same-class samples from different groups.
• Explicitly enforcing embedding similarity between same-class samples from diverse groups, combined with pushing apart hard negatives, is crucial for achieving group robustness with lightweight adapters.

Summary by alphaXiv.