Reproduce “Difference Target Propagation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Dong-Hyun Lee, Saizheng Zhang, Asja Fischer, Yoshua Bengio
Difference Target Propagation offers an alternative credit assignment mechanism for deep networks, addressing back-propagation's limitations with strong non-linearities, discrete computations, and stochastic units. This method demonstrated state-of-the-art performance for stochastic neural networks on MNIST and achieved results comparable to back-propagation on standard continuous networks.
Key insights
- DTP's linear correction provides a crucial stability property: if a layer's current activation reaches its target, the target for the preceding layer will also match its current activation, preventing further unnecessary updates.
- Theoretical analysis demonstrates that DTP's update direction is closely aligned (angle bounded by 90 degrees) with the back-propagation gradient direction, ensuring global loss reduction under specific conditions for approximate inverse mappings.
- The method broadens the scope of trainable deep learning architectures to include those with extreme non-linearities, discrete signal transmissions, and stochastic units, which are typically challenging for back-propagation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.