Reproduce “Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architectures” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Julien Launay, Iacopo Poli, François Boniface, Florent Krzakala

Researchers demonstrated that Direct Feedback Alignment (DFA) can effectively train advanced deep learning models across various contemporary tasks, including neural view synthesis and Transformer-based language modeling, often achieving performance close to backpropagation when appropriately tuned. This work refutes prior assumptions about DFA's scalability and potential for architectures without synaptic symmetry.

Key insights

  • Learning under synaptic asymmetry is feasible for complex deep learning tasks and modern architectures, directly contradicting previous findings about the limitations of DFA.
  • DFA's inherent capability for asynchronous weight updates can enable significant computational speedups and facilitates the development of specialized hardware co-processors (e.g., optical computing).
  • Achieving optimal performance with DFA often requires specific hyperparameter tuning and a rethinking of common deep learning practices, rather than simply adopting settings optimized for Backpropagation.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.