Reproduce “Nonlinear Meta-Learning Can Guarantee Faster Rates” on OpenResearch

About this paper

Dimitri Meunier, Zhu Li, Arthur Gretton, Samory Kpotufe

The paper studies few-data regression on a task by estimating a nonlinear feature space from source tasks, then fitting the target within it. In an RKHS, cross-products of regularized kernel fits estimate the subspace. Under conditions, target risk scales as √(s/n_T) plus subspace error. With 250 source tasks, simulation matches an oracle, but this is not a real-data demonstration.

Key insights

• Optimal meta-learning rates in nonlinear settings are achieved through a deliberate strategy of 'under-regularization' (effectively, overfitting) on individual source tasks, which is crucial for minimizing bias in the estimated shared representation.
• The meta-learning approach can achieve a parametric rate of `O(√(s/n_T))` for the target task's excess risk, demonstrating faster convergence than typical nonparametric rates by effectively learning the shared s-dimensional representation.
• The interplay between the kernel and data distribution, characterized by a parameter `α`, plays a direct and more explicit role in determining meta-learning rates and `N`-saturation points compared to its influence in standard kernel regression.

Summary by alphaXiv.