Reproduce “Predictive Coding Approximates Backprop along Arbitrary Computation Graphs” on OpenResearch

About this paper

Beren Millidge, Alexander Tschantz, Christopher L. Buckley

Researchers demonstrated that predictive coding, a biologically plausible framework, can approximate automatic differentiation and thus backpropagation gradients for arbitrary computation graphs. The method achieved performance equivalent to standard backprop on deep learning models like CNNs, RNNs, and LSTMs, utilizing only local and Hebbian learning rules.

Key insights

• Predictive coding, when properly generalized, can rapidly and robustly approximate exact backpropagation gradients for any differentiable computation graph.
• The derived learning rules are entirely local, meaning each unit updates its state and parameters based only on information from its immediate neighbors, resolving the 'weight transport problem' and non-local information issues of backprop.
• The recursive computation of gradients in backpropagation can be fundamentally understood as a byproduct of a variational inference algorithm minimizing free-energy over the computation graph.

Summary by alphaXiv.