Reproduce “Small Data Challenges in Big Data Era: A Survey of Recent Progress on Unsupervised and Semi-Supervised Methods” on OpenResearch

About this paper

Guo-Jun Qi, Jiebo Luo

This survey systematically reviews recent advancements in unsupervised and semi-supervised deep learning methods, addressing the challenge of limited labeled data by leveraging readily available unlabeled information. It categorizes diverse approaches and outlines foundational principles such as transformation equivariance, disentangled representations, and self-supervision, demonstrating their increasing practical efficacy across various tasks and datasets.

Key insights

• Transformation equivariance, disentangled representations, and self-supervision are critical underlying principles for learning robust features from unlabeled data, enabling generalization beyond direct supervision.
• Teacher-student models, particularly those using consistency regularization (e.g., Mean Teacher, VAT, MixMatch), are highly effective in semi-supervised settings, achieving state-of-the-art results with minimal labeled data.
• Unsupervised pre-training, especially with methods like contrastive learning, can achieve competitive or superior performance to supervised models on cross-dataset tasks, making it a viable foundation for downstream tasks.

Summary by alphaXiv.