Reproduce “Understanding Adversarial Transfer: Why Representation-Space Attacks Fail Where Data-Space Attacks Succeed” on OpenResearch

About this paper

Isha Gupta, Rylan Schaeffer, Joshua Kazdan, Ken Ziyu Liu, Sanmi Koyejo

This research clarifies adversarial transferability by distinguishing between data-space and representation-space attacks, explaining why data-space attacks consistently transfer while representation-space attacks only transfer under conditions of geometric alignment in latent spaces. It provides a unified explanation for previously conflicting observations across image classifiers, language models, and vision-language models.

Key insights

• Adversarial attacks operating directly on the input data-space exhibit robust transferability across functionally similar models, as their input-output mappings are often consistent.
• Attacks optimized within a model's internal representation-space typically lack transferability due to the unique, high-dimensional geometric structure of latent spaces across different models, even those with identical architectures or functionalities.
• Representation-space attacks can achieve transferability if the latent spaces of the source and target models are geometrically aligned, as evidenced by finetuned variants of the same base model exhibiting high representational similarity.

Summary by alphaXiv.