Reproduce “Accelerating Molecular Graph Neural Networks via Knowledge Distillation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Filip Ekström Kelvinius, Dimitar Georgiev, Artur Petrov Toshev, Johannes Gasteiger
Researchers investigated Knowledge Distillation (KD) to enhance the predictive accuracy of molecular Graph Neural Networks (GNNs) for energy and force predictions while maintaining their inference throughput. Custom feature-based KD strategies improved student model performance, consistently closing the accuracy gap by 60% or more for energy predictions across various configurations.
Key insights
- Feature-based KD is highly effective for molecular GNNs, successfully transferring knowledge even between models with significant architectural differences.
- The choice of distillation loss (MSE) and transformation function (linear layer) for features significantly impacts KD performance, with MSE and linear layers proving optimal.
- Distilling knowledge from later layers of the teacher model into deeper layers of the student generally yields better results, suggesting the transfer of more refined, task-relevant information.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.