Reproduce “On the Principles of Parsimony and Self-Consistency for the Emergence of Intelligence” on OpenResearch

About this paper

Yi Ma, Doris Tsao, Heung-Yeung Shum

Researchers present Parsimony and Self-consistency as fundamental principles governing intelligence, showing how their computational application via rate reduction can derive deep network architectures and integrate disparate learning paradigms. This framework offers an interpretable explanation for deep learning's efficacy, resolving issues like neural collapse, catastrophic forgetting, and mode collapse.

Key insights

• Deep network architectures like ResNets, ResNeXts, CNNs, and Transformers can be derived from the iterative optimization steps for maximizing rate reduction, providing a 'white-box' understanding of their structure and success.
• The proposed framework unifies discriminative and generative models, as well as supervised, unsupervised, and incremental learning paradigms, into a single, coherent computational system.
• Intelligence emerges from the interplay of minimizing internal discrepancy (self-consistency) and organizing observations into the most compact, structured representations (parsimony).

Summary by alphaXiv.