Reproduce “The Forward-Forward Algorithm: Some Preliminary Investigations” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Geoffrey Hinton
The Forward-Forward algorithm trains networks with two forward passes: real examples increase each layer’s goodness, while negative examples decrease it. Local activity scores guide layers, and hidden-vector normalization passes activity patterns onward rather than earlier goodness scores. On CIFAR-10, the two-hidden-layer test error was 41% for label-by-label scoring, versus 37% for backpropagation; these small-network results establish neither speed nor energy savings.
Key insights
- The squared norm of hidden layer activations (||h||²) functions effectively as a layer-wise "goodness" score for guiding learning.
- Layer normalization is critical to ensure the network learns meaningful features by preventing it from distinguishing positive and negative data based solely on the length of activity vectors.
- Generating effective negative examples, such as "hybrid images" created by combining parts of different training examples, is crucial for the algorithm's performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.