Reproduce “Abelian Neural Networks” on OpenResearch

About this paper

Kenshin Abe, Takanori Maehara, Issei Sato

This work introduces Abelian Neural Networks (AGN and ASN), a new class of architectures that use invertible neural networks to learn binary operations inherently satisfying algebraic properties like commutativity and associativity. The models demonstrate superior size generalization, maintaining performance on larger inputs, and show improved accuracy on tasks such as word analogies.

Key insights

• A function over multisets is permutation invariant and well-defined if and only if the underlying binary operation forms an Abelian semigroup.
• By transforming inputs to a latent space via an invertible function where simple arithmetic operations (addition or multiplication) are applied, complex algebraic properties like associativity and commutativity can be inherently satisfied and preserved.
• Invertible Neural Networks (INNs) can serve as the trainable invertible function, enabling universal approximation of Abelian Lie group operations and ensuring size generalization for multiset inputs.

Summary by alphaXiv.