Reproduce “Per-Example Gradient Regularization Improves Learning Signals from Noisy Data” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Xuran Meng, Yuan Cao, Difan Zou

Researchers at the University of Hong Kong theoretically and empirically demonstrated that Per-Example Gradient Regularization (PEGR) enables two-layer convolutional neural networks to effectively learn true signals from noisy data while suppressing noise memorization, leading to superior generalization and robustness. The study explains that PEGR achieves this by implicitly controlling the variance of gradients, selectively penalizing noise more strongly than signal.

Key insights

  • Per-Example Gradient Regularization (PEGR) implicitly controls the variance of gradients across different training data points, which selectively suppresses noise memorization more strongly than signal learning.
  • PEGR's advantages in promoting signal learning are most pronounced during the early training stages, particularly when the signal-to-noise ratio in the data is low.
  • The analysis provides a theoretical framework for determining an appropriate time to deactivate the gradient regularization, preventing over-regularization while ensuring optimal performance.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.