Reproduce “Barriers for Learning in an Evolving World: Mathematical Understanding of Loss of Plasticity” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Amir Joudaki, Giulia Lanzillotta, Mohammad Samragh Razlighi, Iman Mirzadeh, Keivan Alizadeh, Thomas Hofmann, Mehrdad Farajtabar, Fartash Faghri

The paper explains how gradient updates can constrain learning through invariant parameter sets. It proves that, under stated conditions, saturated and cloned units remain on lower-dimensional manifolds, and shows SGD preserves cloning while noise or dropout can break it. In a bit-flipping task, switching to CBP accompanies recovery in loss and effective rank, not universal escape.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.