Reproduce “Semi-Supervised Learning via Weight-aware Distillation under Class Distribution Mismatch” on OpenResearch

About this paper

Pan Du, Suyun Zhao, Zisen Sheng, Cuiping Li, Hong Chen

Researchers from Renmin University of China developed Weight-Aware Distillation (WAD), a semi-supervised learning framework that effectively addresses class distribution mismatch by using robust representations and adaptive weighting for pseudo-labeling and filtering unknown categories. WAD consistently outperformed state-of-the-art methods, showing stable accuracy gains (e.g., 2.91% to 4.1% on CIFAR10) and robustness to high proportions of unknown data and numerous unseen classes.

Key insights

• Semi-supervised learning (SSL) error, particularly under class distribution mismatch, comprises both pseudo-labeling error for target instances and "invasion error" from unknown categories.
• Robust representations, learned through unsupervised contrastive methods, can serve as a superior teacher to generate high-quality pseudo-labels, avoiding the biases of an immature student classifier.
• Adaptive instance weighting, determined by the confidence and ambiguity of pseudo-labels, is effective in filtering out unknown categories and managing instances near decision boundaries.

Summary by alphaXiv.