Reproduce “Learning Dynamics of Deep Learning -- Force Analysis of Deep Neural Networks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yi Ren
The authors present force analysis to trace how one example’s update changes another’s prediction. It separates prediction-to-supervision gaps, gradient-space interactions, and current-prediction effects. It applies this lens to distillation, language-model finetuning, hidden features, and compositional learning. In CIFAR-100 ResNet18 self-distillation, Filter-KD reaches 80.09% accuracy and 0.029 ECE, versus 78.83% and 0.067 for ESKD, in the reported setup.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.