Reproduce “Compressing Neural Networks with the Hashing Trick” on OpenResearch

About this paper

Wenlin Chen, James T. Wilson, Stephen Tyree, Kilian Q. Weinberger, Yixin Chen

HashedNets, a new neural network architecture from Washington University in St. Louis and NVIDIA, employs the hashing trick for random weight sharing to drastically reduce model size. This method allows for larger virtual network structures with fewer actual parameters, consistently outperforming existing compression techniques, especially at high compression ratios.

Key insights

• The hashing trick, traditionally used for features, can be effectively applied to neural network weights to achieve substantial memory compression without explicit storage of sharing assignments.
• Neural network weights exhibit significant inherent redundancy, enabling aggressive sharing of parameters while maintaining model expressiveness and accuracy.
• Maintaining a larger 'virtual' network structure with shared weights can yield better generalization and performance than a physically smaller, fully parameterized network with the same number of unique parameters.

Summary by alphaXiv.