Reproduce “Learning to Reweight Examples for Robust Deep Learning” on OpenResearch

About this paper

Mengye Ren, Wenyuan Zeng, Bin Yang, Raquel Urtasun

Researchers from Uber Advanced Technologies Group and the University of Toronto developed an online meta-learning algorithm to dynamically reweight training examples. The method leverages a small, clean validation set to guide the optimization, achieving superior robustness against class imbalance and label noise across standard benchmarks.

Key insights

• A small, unbiased validation set can effectively guide an online reweighting mechanism, enabling robust training without relying on complex heuristics or assumptions about noise types.
• Framing reweighting as a meta-learning problem allows the model to learn optimal example importance by considering how each example's gradient impacts performance on a clean validation objective.
• The technique efficiently identifies and down-weights noisy or harmful training examples based on their gradient alignment with the validation loss, leading to less overfitting and better generalization.

Summary by alphaXiv.