Reproduce “WERank: Towards Rank Degradation Prevention for Self-Supervised Learning Using Weight Regularization” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ali Saheb Pasand, Reza Moravej, Mahdi Biparva, Ali Ghodsi
Researchers from the University of Waterloo, University of Toronto, and Huawei Noah ’s Ark Lab developed WERank, a weight regularization technique to prevent dimensional collapse in self-supervised learning by directly regularizing network weights. This method ensures representations maintain high rank across all layers, leading to improved downstream node classification accuracy on graph datasets when integrated with existing SSL methods like BGRL.
Key insights
- Rank degradation in deep networks can occur in early layers and propagate, rendering final-layer regularization insufficient for full collapse prevention.
- Regularizing weight matrices to encourage their singular values towards one can fundamentally prevent dimensional collapse at each layer by promoting orthonormal transformations.
- This weight-based approach can yield more stable, distance-preserving "short maps" across network layers, enhancing representation robustness.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.