Reproduce “Non-Stationary Learning of Neural Networks with Automatic Soft Parameter Reset” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Alexandre Galashov, Michalis K. Titsias, András György, Clare Lyle, Razvan Pascanu, Yee Whye Teh, Maneesh Sahani

The paper introduces Soft Reset, which adapts a neural network to changing data by estimating how strongly parameters should drift toward their initialization. An Ornstein–Uhlenbeck process sets this drift and raises learning rate, while a Bayesian variant updates uncertainty. Across streams, Soft Reset beats Hard Reset and most baselines, especially when changes are less frequent; results are empirical, not guaranteed.

Key insights

  • The Ornstein-Uhlenbeck process effectively interpolates network parameters between their current state and initialization, preserving plasticity over time.
  • The learned drift parameter (γ_t) adaptively controls the degree of soft reset, decreasing when non-stationarity is detected (e.g., at task boundaries or with large gradients) to increase the effective learning rate.
  • Non-stationarity in off-policy reinforcement learning primarily affects value function networks, as indicated by layer-specific changes in γ_t, rather than the policy network.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.