Reproduce “Deep Classifier Mimicry without Data Access” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Steven Braun, Martin Mundt, Kristian Kersting
CAKE (Contrastive Abductive Knowledge Extraction) enables model-agnostic knowledge distillation without access to original training data, focusing on mimicking the teacher's decision boundary. The method achieves competitive performance compared to data-dependent techniques and significantly outperforms other data-free, model-agnostic approaches like DeepDream by over 2x on CIFAR-10.
Key insights
- For supervised classification knowledge distillation, the primary goal is to accurately mimic the teacher's decision boundary, not necessarily to reconstruct the full original data distribution.
- A combination of contrastive loss, strategic noise, and optional domain priors can effectively generate synthetic samples that map a black-box model's decision boundary without accessing original data or internal architecture.
- The synthetic samples generated by CAKE do not visually resemble original training data, suggesting potential for privacy-preserving knowledge transfer.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.