Reproduce “Orthogonal Projection Loss” on OpenResearch

About this paper

Kanchana Ranasinghe, Muzammal Naseer, Munawar Hayat, Salman Khan, Fahad Shahbaz Khan

Researchers from MBZUAI, ANU, Monash, and Linköping Universities introduce Orthogonal Projection Loss (OPL), a loss function that enhances deep feature discrimination by enforcing inter-class orthogonality and intra-class clustering. OPL improves performance across classification, label noise robustness, adversarial attack robustness, domain generalization, and few-shot learning tasks, including a 0.83% increase in ImageNet ResNet-50 top-1 accuracy and a 1.01% improvement in average accuracy on PACS for domain generalization.

Key insights

• Enforcing orthogonality in the feature space directly complements the intrinsic angular properties of softmax CE loss, leading to more geometrically structured and discriminative features.
• OPL ensures inter-class independence and intra-class compactness with a simple, 'plug-and-play' formulation that avoids common pitfalls of other advanced loss functions.
• The learned representations are shown to be more robust and generalizable across various challenging computer vision tasks, including those with noisy data, adversarial examples, and domain shifts.

Summary by alphaXiv.