Reproduce “LLMs Are Few-Shot In-Context Low-Resource Language Learners” on OpenResearch

About this paper

Samuel Cahyawijaya, Holy Lovenia, Pascale Fung

Researchers from Hong Kong University of Science and Technology and AI Singapore conducted an extensive study on In-Context Learning (ICL) and Cross-Lingual ICL (X-ICL) for 25 low-resource and 7 higher-resource languages using XGLM-7.5B and BLOOM-7B. They introduced in-context query alignment, which improved performance by approximately 10% Weighted F1 in 56.25% of cases, and provided practical guidelines for leveraging Large Language Models in data-scarce linguistic contexts.

Key insights

• In-context label alignment generally degrades performance in low-resource settings (by ~20% Weighted F1 in 88.46% of cases), challenging previous findings, likely due to weak label representations.
• The proposed in-context query alignment is a more effective strategy, improving performance by aligning input distributions without shifting the label space, offering a robust alternative for low-resource languages.
• For low-resource languages, improving semantic representations through X-ICL or query alignment is more crucial than subtle prompt formatting, and semantic retrieval of exemplars consistently outperforms random selection.

Summary by alphaXiv.