Reproduce “Globally Injective ReLU Networks” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Michael Puthawala, Konik Kothari, Matti Lassas, Ivan Dokmanić, Maarten de Hoop

This research provides a mathematical framework for understanding and constructing globally injective ReLU neural networks by establishing necessary and sufficient conditions for injectivity in individual layers and deep architectures. It demonstrates that generative models built with these principles improve latent space inference without sacrificing generative quality.

Key insights

  • Global injectivity of a ReLU layer is precisely characterized by its weight matrix having a Directed Spanning Set with respect to every input vector.
  • Injective ReLU networks are universal approximators for any continuous function, indicating that enforcing injectivity does not inherently limit network expressivity.
  • An output dimension of at least $2n$ (where $n$ is the input dimension) is a critical factor for achieving global injectivity in ReLU layers and deep networks.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.