Reproduce “Diversifying Counterattacks: Orthogonal Exploration for Robust CLIP Inference” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Chengze Jiang, Minjing Dong, Xinli Shi, Jie Gui

Directional Orthogonal Counterattack (DOC) improves the adversarial robustness of CLIP models by generating more diverse counterattack perturbations through orthogonal exploration and an adaptive directional sensitivity score. This approach increased robust accuracy by 9.80% over previous state-of-the-art test-time defenses under PGD-10 attacks, maintaining strong performance across various attack types and preserving clean accuracy.

Key insights

  • Increasing the diversity and coverage of counterattack perturbations is crucial for enhancing adversarial robustness in test-time defenses for CLIP models.
  • Relying solely on a narrow gradient-based objective (like maximizing embedding distance) restricts the counterattack search space, leading to overfitting and limited generalizability.
  • An adaptive mechanism, such as a directional sensitivity score, can improve the discrimination between clean and adversarial examples, allowing for modulated counterattack strength without degrading clean accuracy.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.