Reproduce “Temporal-Difference Learning Using Distributed Error Signals” on OpenResearch

About this paper

Jonas Guan, Shon Eduard Verch, Claas Voelcker, Ethan C. Jackson, Nicolas Papernot, William A. Cunningham

A novel deep Q-learning algorithm, ARTIFICIAL DOPAMINE (AD), demonstrates that synchronously distributed, per-layer temporal-difference error signals are sufficient for neural networks to learn complex reward-based tasks. The algorithm achieves performance comparable to backpropagation-based methods across various reinforcement learning benchmarks and offers a biologically plausible model for credit assignment in the nucleus accumbens.

Key insights

• Synchronously distributed, local TD error signals, without explicit error propagation, are computationally sufficient for deep reinforcement learning tasks, aligning with biological observations in the NAc.
• Forward-in-time downward connections, enabling higher-level activations to influence lower layers at the next timestep, are crucial for effective multi-layer coordination and can effectively substitute for error backpropagation.
• Multi-layer architectures are critical for robust performance, consistently outperforming even widened single-layer configurations, indicating that inter-layer coordination drives observed capabilities.

Summary by alphaXiv.