Reproduce “Practical Black-Box Attacks against Machine Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z. Berkay Celik, Ananthram Swami
A black-box attack method effectively generates adversarial examples for remotely hosted machine learning models, achieving misclassification rates up to 96.19% against commercial APIs using a local substitute model with a limited query budget. This approach also successfully evades gradient masking defenses, highlighting practical vulnerabilities in deployed AI systems.
Key insights
- Black-box machine learning models are practically vulnerable to adversarial attacks, requiring minimal adversary knowledge (only query access to labels) and a limited query budget.
- Adversarial examples crafted for a local substitute model can effectively 'transfer' and fool a distinct black-box target model, even with different architectures, due to high correlation in cost gradient signs.
- Defenses relying on 'gradient masking' (e.g., certain adversarial training and defensive distillation) are insufficient against black-box transferability attacks, as the substitute model can still compute clear gradients.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.