Reproduce “Forward Only Learning for Orthogonal Neural Networks of any Depth” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Paul Caillon, Alex Colagrande, Erwan Fagnou, Blaise Delattre, Alexandre Allauzen

FOTON, a novel forward-only learning algorithm, successfully enables the training of deep and convolutional neural networks without relying on backpropagation. It demonstrates competitive performance with backpropagation across various architectures and datasets while significantly reducing memory overhead by eliminating the backward pass.

Key insights

  • Orthogonality of weight matrices is crucial for maintaining a stable and consistent error signal transmission through numerous layers in forward-only neural networks, enabling training of arbitrary depth.
  • FOTON's update directions consistently align positively with backpropagation's true gradients in non-linear networks, overcoming the gradient misalignment issues observed in prior forward-only methods.
  • Frequent orthogonalization of weight matrices is paramount for the stability and effective performance of deep FOTON networks, with less frequent updates leading to performance degradation in deeper models.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.