Reproduce “Convergent Emergence of In-Context Learning Across Modalities” on OpenResearch

About this paper

Nathan Breslow, Seungwook Han, Daniel Hyunsoo Lee, Aayush Mishra, Anqi Liu, Daniel Khashabi

A unified eight-bit function benchmark shows that paired-demonstration in-context learning emerges in language, genomic and protein sequences, integer sequences, images, and time series without explicit meta-ICL training. Correct pairings improve maximum-shot accuracy by 9.9–31.4 percentage points across these modalities, while task-level profiles converge strongly among five modalities but diverge for images; chess shows no measurable effect and music only a weak one.

Key insights

• Paired-mapping ICL appears across six diverse modalities, indicating that explicit few-shot supervision is not necessary for models to extract abstract input-output rules from demonstrations.
• Five modalities show positive correlations in which functions benefit most from correct pairings, with genome and protein profiles correlating at 0.89; ImageGPT exhibits ICL but a substantially different task profile.
• Model scale generally strengthens ICL but is not sufficient or uniformly predictive: some larger genomic and protein models underperform smaller counterparts, while chess and music show little or no detectable effect.

Summary by alphaXiv.