Reproduce “Predicting Parameters in Deep Learning” on OpenResearch

About this paper

Misha Denil, Babak Shakibi, Laurent Dinh, Marc'Aurelio Ranzato, Nando de Freitas

The paper by Denil et al. demonstrates that a substantial portion of deep learning model parameters are redundant and can be accurately predicted from a smaller learned subset. This method significantly reduces the number of dynamic, trainable parameters, maintaining high predictive accuracy across MLPs, ConvNets, and RICA models, and in some cases, allows for more expressive models under a fixed budget of learnable parameters.

Key insights

• Deep learning models exhibit a significant degree of parameter redundancy, allowing a large proportion of weights (up to 95%) to be accurately predicted from a much smaller learned subset without loss of accuracy.
• Carefully constructed dictionaries, informed by prior knowledge (e.g., kernels for spatial smoothness) or data-driven insights (e.g., from autoencoders), are crucial for effective parameter prediction and maintaining model performance.
• The parameter prediction technique enables the creation of richer, more expressive models for a fixed budget of dynamic (learnable) parameters, potentially enhancing predictive accuracy compared to standard models with the same learnable parameter count.

Summary by alphaXiv.