Reproduce “Switch-KD: Visual-Switch Knowledge Distillation for Vision-Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Haoyi Sun, Xiaoxiao Wang, Ning Mao, Qian Wang, Lifu Mu, Wen Zheng, Tao Wei, Wei Chen

Researchers at Li Auto Inc. developed Switch-KD, a knowledge distillation framework for Vision-Language Models that unifies visual and language knowledge transfer in a shared text-probability space. It employs a Visual-Switch Distillation mechanism and a Dynamic Bi-directional Logits Difference (DBiLD) loss, achieving average performance gains of 4.9 and 3.6 points for 0.5B student models over baselines and surpassing existing state-of-the-art VLM distillation methods across multiple benchmarks.

Key insights

  • Implicit visual knowledge transfer can be effectively achieved by aligning the student's visual encoder output with the teacher's robust language pathway, fostering stronger visual-semantic alignment without explicit visual feature matching.
  • Dynamically focusing on the most informative parts of VLM logits distributions via an adaptive top-k selection and bidirectional reverse KL divergence significantly enhances distillation efficacy and stability for long-tailed outputs.
  • Distillation applied during the fine-tuning stage (Distilled Fine-Tuning) is highly effective for VLM knowledge transfer, yielding substantial performance gains for student models compared to other distillation strategies.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.