Reproduce “Learn to Preserve and Diversify: Parameter-Efficient Group with Orthogonal Regularization for Domain Generalization” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jiajun Hu, Jian Zhang, Lei Qi, Yinghuan Shi, Yang Gao

Researchers from Nanjing University and Southeast University introduce PEGO, a Parameter-Efficient Group with Orthogonal regularization method that enhances domain generalization for foundation models. PEGO achieves state-of-the-art performance on five DomainBed benchmarks with an average accuracy of 76.1%, outperforming prior methods while efficiently preserving pre-trained knowledge and diversifying learned features.

Key insights

  • Explicitly enforcing orthogonality between pre-trained weights and fine-tuning adaptation weights effectively preserves the generalized knowledge of foundation models during adaptation.
  • Applying orthogonal regularization within a group of low-rank adaptation modules promotes the learning of diverse feature representations, which is crucial for handling wide distribution shifts in domain generalization.
  • Combining parameter efficiency with a dual strategy of knowledge preservation and diversification offers a robust and effective approach to fine-tune large foundation models for superior out-of-distribution performance without incurring additional inference latency.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.