Reproduce “In-Context Learning with Long-Context Models: An In-Depth Exploration” on OpenResearch

About this paper

Amanda Bertsch, Maor Ivgi, Emily Xiao, Uri Alon, Jonathan Berant, Matthew R. Gormley, Graham Neubig

This paper thoroughly investigates in-context learning (ICL) at extreme scales, providing thousands of demonstrations to large language models with extended context windows. It demonstrates that performance on various tasks consistently improves with more in-context examples, often approaching or exceeding parameter-efficient finetuning, and reveals that these gains are largely due to the increased availability of examples for "retrieval" rather than complex inter-demonstration contextualization.

Key insights

• The primary benefit of long-context in-context learning (ICL) comes from the increased number of relevant examples available for the model to attend to, acting more like an implicit retrieval mechanism.
• Deep, long-range attention across the entire set of in-context demonstrations is not the main driver of performance gains; local attention to relevant examples appears sufficient.
• ICL becomes less sensitive to random example ordering with more demonstrations, but performance is negatively impacted when examples are explicitly grouped by label, suggesting models benefit from interleaved diversity.

Summary by alphaXiv.