Reproduce “AuPair: Golden Example Pairs for Code Repair” on OpenResearch

About this paper

Aditi Mavalankar, Hassan Mansoor, Zita Marinho, Masha Samsikova, Tom Schaul

AuPair, developed by Google DeepMind, enhances large language model code repair by leveraging strategically selected "golden example pairs" for in-context learning. This approach consistently outperforms best-of-N and self-repair baselines, demonstrating strong generalization across various models and datasets while utilizing inference-time compute efficiently.

Key insights

• Strategic selection of in-context examples through a submodular process yields significantly higher compute efficiency and performance gains than random examples or verbal feedback for code repair.
• The method demonstrates robust generalization of learned repair patterns, with AuPairs from one dataset effectively improving performance on diverse out-of-distribution coding problems.
• AuPair effectively mitigates initial instruction-following issues and consistently scales performance with increased inference-time compute, unlike conventional baselines.

Summary by alphaXiv.