Reproduce “Understanding Representation of Deep Equilibrium Models from Neural Collapse Perspective” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Haixiang Sun, Ye Shi
Researchers from ShanghaiTech University analyzed the representation of Deep Equilibrium Models (DEQs) through the lens of Neural Collapse (NC), demonstrating that DEQs achieve a lower loss bound and exhibit superior feature learning for minority classes in imbalanced datasets compared to explicit neural networks.
Key insights
- Neural Collapse (NC1, NC2, NC3) phenomena are observed in Deep Equilibrium Models, extending the theoretical understanding of NC to implicit architectures.
- Deep Equilibrium Models theoretically achieve a lower lower-bound on the loss function compared to explicit neural networks, indicating stronger learning potential.
- DEQ's iterative, fixed-point solving nature provides more 'adjustment space' for parameters, enhancing feature learning and adaptability for minority classes in imbalanced settings.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.