Reproduce “Understanding and Mitigating the Label Noise in Pre-training on Downstream Tasks” on OpenResearch

About this paper

Hao Chen, Jindong Wang, Ankit Shah, Ran Tao, Hongxin Wei, Xing Xie, Masashi Sugiyama, Bhiksha Raj

Researchers from Carnegie Mellon University, Microsoft Research Asia, and other institutions investigated how label noise in large-scale pre-training datasets affects downstream task performance. They developed NMTune, a lightweight, black-box fine-tuning method that consistently improves generalization, particularly for out-of-domain tasks, by adaptively reshaping feature representations.

Key insights

• Pre-training label noise has a dual effect: slight noise (e.g., 5-10%) can surprisingly benefit in-domain generalization, while any amount of noise consistently deteriorates out-of-domain robustness.
• Feature space analysis revealed that slight noise increases Singular Value Entropy (SVE) for ID benefits, while all noise increases Largest Singular Value Ratio (LSVR), hindering OOD transferability by weakening dominant feature components.
• It is possible to effectively mitigate the negative impacts of pre-training noise and improve both ID and OOD performance through targeted regularization that reshapes the feature space during lightweight fine-tuning.

Summary by alphaXiv.