Reproduce “Meta Two-Sample Testing: Learning Kernels for Testing with Limited Data” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Feng Liu, Wenkai Xu, Jie Lu, Danica J. Sutherland

Meta-MKL learns how to select kernels from related two-sample tasks, then adapts a combination of those kernels using a small training split from a new task before testing with held-out data. Test power is the proportion of tests rejecting the null when distributions differ. On CIFAR-10 versus CIFAR-10.1, power was 0.728 ± 0.020, but the approach assumes relevant related tasks.

Key insights

  • Knowledge transferred from a distribution of related auxiliary tasks enables the effective learning of a powerful kernel selection algorithm for new, data-limited tasks.
  • The theoretical analysis highlights that task relatedness and the efficiency of adaptation (e.g., few parameters in MKL) are crucial for improved performance in low-data settings.
  • Explicitly learning the convex combination weights for a set of pre-optimized base kernels (Meta-MKL) is often more robust and stable than adapting deep network parameters (Meta-KL) with very limited target data.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.