Reproduce “Gradient Agreement as an Optimization Objective for Meta-Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Amir Erfan Eshratifar, David Eigen, Massoud Pedram
A new optimization method for meta-learning, Gradient Agreement, dynamically weights task contributions in a batch based on their gradient alignment to improve generalization. This approach led to an over 10% relative accuracy increase on miniImageNet for 5-way 1-shot and 5-way 5-shot classification compared to MAML and Reptile.
Key insights
- Not all tasks in a meta-training batch contribute equally effectively to a robust meta-initialization; some tasks' gradients may disagree with the consensus.
- Weighting tasks based on the alignment of their gradients with the collective batch gradient can lead to more stable and effective meta-parameter updates.
- Optimizing for gradient agreement guides the meta-parameters towards directions that are broadly consistent across tasks, improving generalization.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.