Reproduce “Direct Distillation between Different Domains” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jialiang Tang, Shuo Chen, Gang Niu, Hongyuan Zhu, Joey Tianyi Zhou, Chen Gong, Masashi Sugiyama
The Direct Distillation between Different Domains (4Ds) method introduces a one-stage framework for training compact student networks on target domain data using a source-trained teacher network without requiring access to source data. It consistently outperforms two-stage and traditional knowledge distillation and domain adaptation methods across multiple benchmarks, achieving student performance comparable to the larger teacher network itself.
Key insights
- Fourier Transform can effectively decouple domain-specific (amplitude) and domain-invariant (phase) knowledge in deep features for targeted adaptation.
- Concurrently adapting the teacher's domain-specific knowledge while preserving its domain-invariant knowledge, and then distilling both categorical and domain-invariant features to the student, is effective for cross-domain model compression.
- An interactive training loop where the teacher also learns from the student (via KL divergence) creates a more "student-friendly" supervisor, improving knowledge transfer across domains and enabling source-data-agnostic distillation, as demonstrated by researchers from Nanjing University of Science and Technology, RIKEN, and A*STAR Institute for Infocomm Research (I2R).
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.