Reproduce “Barriers for Learning in an Evolving World: Mathematical Understanding of Loss of Plasticity” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Amir Joudaki, Giulia Lanzillotta, Mohammad Samragh Razlighi, Iman Mirzadeh, Keivan Alizadeh, Thomas Hofmann, Mehrdad Farajtabar, Fartash Faghri
The paper explains how gradient updates can constrain learning through invariant parameter sets. It proves that, under stated conditions, saturated and cloned units remain on lower-dimensional manifolds, and shows SGD preserves cloning while noise or dropout can break it. In a bit-flipping task, switching to CBP accompanies recovery in loss and effective rank, not universal escape.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.