Reproduce “Training a Student Expert via Semi-Supervised Foundation Model Distillation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Pardis Taghavi, Tian Liu, Renjie Li, Reza Langari, Zhengzhong Tu
Researchers from Texas A&M University developed a semi-supervised knowledge distillation framework to train compact student models for instance segmentation, leveraging Vision Foundation Models. This approach achieved an 11x reduction in model size and FLOPS, and a 4.5x improvement in inference speed, while outperforming state-of-the-art semi-supervised methods on Cityscapes and ADE20K datasets with limited labeled data.
Key insights
- Adapting the VFM teacher to the target domain before distillation is crucial for generating high-quality pseudo-labels and significantly improving student performance.
- Explicitly encouraging inter-instance separation at the pixel level via a contrastive loss with debiased negative sampling enhances the quality of dense mask predictions, even with limited labeled data.
- A multi-stage distillation approach can effectively mitigate pseudo-label noise and biases introduced during semi-supervised training, leading to sharper decision boundaries in the student model.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.