Reproduce “Iterative Amortized Inference: Unifying In-Context Learning and Learned Optimizers” on OpenResearch

About this paper

Sarthak Mittal, Divyat Mahajan, Guillaume Lajoie, Mohammad Pezeshki

Mittal et al. introduce a unified framework for amortized learning, encompassing meta-learning, in-context learning, and learned optimizers, categorized into parametric, implicit, and explicit regimes. They propose "iterative amortized inference," a scalable approach that refines solutions progressively over mini-batches of task data, consistently enhancing performance across diverse tasks and demonstrating improved efficiency and generalization.

Key insights

• Diverse amortized learning paradigms can be formally unified, clarifying their commonalities and specific mechanisms for encoding and leveraging task-specific information.
• Iterative processing of task data over mini-batches significantly improves the scalability and performance of amortized inference across all defined regimes (parametric, explicit, implicit).
• For many tasks, especially lower-dimensional ones, leveraging raw observations directly during adaptation can be more effective than relying solely on gradients, as seen in parametric models.

Summary by alphaXiv.