Reproduce “On Good Practices for Task-Specific Distillation of Large Pretrained Visual Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Juliette Marrie, Michael Arbel, Julien Mairal, Diane Larlus

The authors present a task-specific distillation recipe that transfers visual knowledge from a pretrained model to a smaller student. They probe a frozen teacher encoder with a task head, then train the student on labeled images and teacher predictions for prompt-free synthetic images. On ADE20K, the student reaches 52.1 mIoU versus 49.8 after fine-tuning, but results vary by dataset.

Key insights

  • Preserving the rich, generic representations of a large pretrained teacher by freezing its backbone (probing) is more effective for knowledge transfer than finetuning the entire teacher model, even if the finetuned teacher achieves higher task accuracy.
  • Effective knowledge transfer can occur even from a teacher model that is less accurate than the finetuned student, indicating that the 'knowledge' conveyed extends beyond mere predictive performance.
  • Direct distillation from very large pretrained models to much smaller students is highly effective, challenging the necessity for intermediate 'teacher assistant' models to bridge capacity gaps.
  • Diffusion-generated synthetic data, when used solely for the distillation loss and not the task loss, significantly improves student performance by providing useful diversity without needing faithful class labels.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.