Reproduce “NeuroPlastic: A Plasticity-Modulated Optimizer for Biologically Inspired Learning Dynamics” on OpenResearch

About this paper

Douglas Jiang, Yuechen Wang, Jiayi Wang, Jiaying Geng, Qinglong Wang, Feng Tian

NeuroPlastic is an optimizer that applies neurobiological principles of multi-factor synaptic plasticity to enhance gradient-based updates in deep learning. It improves performance on benchmarks, especially in low-data and more challenging scenarios, by modulating update signals with combined gradient, activity, and memory traces.

Key insights

• Enriching gradient-based updates with multi-factor plasticity signals can yield consistent performance improvements, particularly in challenging scenarios with limited or noisy learning signals.
• The benefits arise from modifying effective update dynamics and their stability, rather than solely altering raw gradient magnitudes.
• The specific formulation and integration method of plasticity signals are critical for effectiveness, with 'controller-style' modulation proving more robust than naive additive variants.

Summary by alphaXiv.