Reproduce “Error Highways: Scaling Predictive Coding to Very Deep Networks” on OpenResearch

About this paper

Amirhossein Mohammadi, Alexander G. Ororbia

The authors introduce HEP, a predictive-coding method that sends output error directly to selected hidden layers during inference. It changes state inference but retains local synaptic weight updates; the forward path still passes through every layer. On a 128-layer skip-free ReLU MLP on MNIST, HEP reaches 95.8 ± 0.6% accuracy, versus 12.8 ± 2.5% for vanilla PC; evidence is limited to these image tasks and this MLP setup.

Key insights

• HEP separates two depth problems: RMS normalization controls forward-state scaling, while direct error highways provide hidden layers with an actionable supervisory signal early in inference.
• Feedback need not reach every layer: at 32 layers, placing highways every second to fifth layer remains close to dense feedback, but larger gaps increasingly reduce accuracy and stability.
• The approach improves credit assignment without replacing local synaptic learning, although it still depends on broadcast output supervision and additional long-range feedback connections.

Summary by alphaXiv.