Reproduce “The Description Length of Deep Learning Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Léonard Blier, Yann Ollivier
This study empirically measures the data compression capabilities of deep learning models, revealing that prequential coding provides significantly tighter description length bounds compared to traditional variational inference. It demonstrates that deep networks effectively compress training data, aligning with information-theoretic principles for generalization.
Key insights
- Deep neural networks can achieve substantial data compression, even when accounting for model complexity, aligning with Occam's razor when an appropriate encoding scheme is used.
- Prequential coding yields significantly lower description length bounds for deep models compared to variational inference, which was found to be inefficient in providing tight bounds.
- A strong correlation exists between lower prequential codelengths and higher test set accuracy, empirically linking data compression capability with generalization performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.