Reproduce “Origami in N dimensions: How feed-forward networks manufacture linear separability” on OpenResearch

About this paper

Christian Keup, Moritz Helias

The paper "Origami in N Dimensions" presents a geometric interpretation of how feed-forward neural networks, particularly those with ReLU activations, achieve linear separability by "folding" data manifolds into higher dimensions. This mechanism creates bimodal tuning curves in neurons and explains how network depth increases expressivity, linking artificial neural networks to principles of computational neuroscience.

Key insights

• The primary mechanism for linear separability in ReLU networks is "folding" data manifolds in higher dimensions, distinguishing it from other transformations like shear.
• This folding process predicts the emergence of bimodal tuning curves in neurons of trained networks, which can be quantified and are linked to network performance.
• The geometric perspective of folding explains why depth is often more efficient than width for increasing network expressivity and provides connections to mixed selectivity observed in biological neurons.

Summary by alphaXiv.