Reproduce “Self-training Improves Pre-training for Few-shot Learning in Task-oriented Dialog Systems” on OpenResearch

About this paper

Fei Mi, Wanhao Zhou, Fengyu Cai, Lingjing Kong, Minlie Huang, Boi Faltings

This research from Huawei Noah’s Ark Lab, EPFL, and Tsinghua University introduces a self-training framework with a novel gradient-based text augmentation technique (GradAug) to improve few-shot performance in task-oriented dialog systems. The method consistently yields substantial performance gains across various dialog tasks, effectively leveraging unlabeled data to narrow the gap between models trained with limited and full supervision.

Key insights

• Self-training provides complementary benefits atop pre-trained models, enhancing performance beyond pre-training alone in few-shot settings.
• The gradient-based text augmentation (GradAug) is crucial for robust student training and outperforms other augmentation methods by preserving semantic fidelity.
• Leveraging unlabeled data via self-training can substantially narrow the performance gap between few-shot models and those trained with full supervision, sometimes matching or exceeding customized domain-specific pre-training.

Summary by alphaXiv.