Reproduce “Awakening of the Buddha: Subspace Learning During Population-Loss Plateaus” on OpenResearch

About this paper

Akash Kumar

The study shows that neural networks can acquire useful representations while their population loss remains nearly unchanged. For specified ReLU and leaky-ReLU settings, training recovers a teacher’s low-dimensional subspace and improves predictions from the learned features before the original predictor’s loss later falls; SwiGLU results establish a narrower form of early alignment.

Key insights

• Representation learning, predictive information in features, and loss reduction are distinct events that can occur at different times during training.
• In the main ReLU result, these events occur along one trajectory: the network recovers the teacher subspace and improves a same-budget feature refit during a near-unit-loss plateau, then its original predictor’s loss decreases later.
• The scope of the guarantees depends on architecture and diagnostic: the main ReLU results cover structured teachers and full subspace recovery, while the broader SwiGLU result concerns leading-direction alignment, not full higher-rank recovery.

Summary by alphaXiv.