Reproduce “An empirical study of domain-agnostic semi-supervised learning via energy-based models: joint-training and pre-training” on OpenResearch

About this paper

Yunfu Song, Huahuan Zheng, Zhijian Ou

This empirical study investigates semi-supervised learning using Energy-Based Models (EBMs), systematically comparing joint-training and pre-training approaches across image classification and natural language labeling tasks. EBM-based joint-training consistently outperforms pre-training and supervised baselines, achieving comparable performance to fully supervised models with up to 50% less labeled data in image tasks and showing improvements in 25 out of 27 NLP settings.

Key insights

• EBM-based joint-training consistently proves more effective for SSL than EBM-based pre-training, suggesting that joint optimization of observations and labels yields more discriminative representations.
• The generative loss component in EBM-based joint-training provides a beneficial regularization effect, improving performance even when abundant labeled data is available.
• The optimal performance gains from EBM-based SSL are task-dependent and generally occur when the supervised baseline performance is in a moderate range, with larger unlabeled data pools enhancing effectiveness.

Summary by alphaXiv.