Reproduce “Two $\operatorname{ReLU}$ Layers is All You Need” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Florestan Brunck, Christoph Hertrich, Kilian Ruess, Jack Stade, Amir Yehudayoff

The authors prove that every continuous piecewise-linear function representable by a ReLU network can be represented with two hidden layers in every input dimension. Their proof reduces the claim to representing the maximum of n inputs with rational parameters. Thus deeper networks add no exactly representable functions in this class, but the theorem gives no bound on width or efficiency.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.