Reproduce “VNE: An Effective Method for Improving Deep Representation by Manipulating Eigenvalue Distribution” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jaeill Kim, Suhyun Kang, Duhun Hwang, Jungwook Shin, Wonjong Rhee
Researchers from Seoul National University developed von Neumann entropy (VNE) as a regularization technique to precisely control the eigenvalue distribution of deep representations, yielding improved performance and stability across diverse deep learning tasks including self-supervised learning, domain generalization, meta-learning, and GANs.
Key insights
- VNE offers superior stability and direct control over eigenvalue distributions compared to Frobenius norm regularization, avoiding issues like dead neurons and ensuring stable optimization.
- The approach establishes formal theoretical connections between VNE and crucial representation properties, serving as a continuous proxy for rank, disentanglement, and isotropy.
- Optimal manipulation of eigenvalue distribution is task-dependent; VNE$^+$ is beneficial for tasks requiring rich, diverse features (e.g., meta-learning, SSL), while VNE$^-$ aids tasks benefiting from reduced effective dimensionality (e.g., domain generalization).
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.