Reproduce “Knowledge Evolution in Neural Networks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ahmed Taha, Abhinav Shrivastava, Larry Davis
A method called Knowledge Evolution (KE) enables effective training of deep neural networks on limited datasets while simultaneously reducing inference costs. This evolutionary approach improves classification performance by up to 21% on datasets such as Flowers-102 and yields slim networks with up to 73% lower inference costs.
Key insights
- Transferring knowledge across generations of a neural network, combined with strategic reinitialization, enables more efficient learning from limited data and helps prevent overfitting.
- Kernel-Level Convolutional-Aware Splitting (KELS) effectively identifies and preserves a coherent subnetwork (fit-hypothesis) suitable for both performance improvement and inference cost reduction.
- The evolutionary knowledge transfer mechanism is crucial for performance gains, outperforming random reinitialization of network parts, particularly on smaller datasets.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.