Reproduce “Effective Demonstration Annotation for In-Context Learning via Language Model-Based Determinantal Point Process” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Peng Wang, Xiaobin Wang, Chao Lou, Shengyu Mao, Pengjun Xie, Yong Jiang
LM-DPP provides an efficient method for annotating in-context learning demonstrations, achieving average accuracy gains of 1.15% on NLU tasks and improving Rouge-L F1 scores by 2.18% on summarization, by optimally balancing uncertainty and diversity with minimal human labeling.
Key insights
- Optimal ICL performance is achieved by balancing both uncertainty and diversity in demonstration selection, with intermediate 'λ' values (typically 0.5-0.6) yielding the best results.
- Smaller LLMs (e.g., GPT2) can effectively estimate perplexity for larger inference models (e.g., GPT-J, LlaMA-2), significantly reducing the computational cost of demonstration selection.
- The LM-DPP approach demonstrates strong transferability across various LLMs, including GPT-3.5-Turbo, and offers enhanced stability and time efficiency compared to existing baselines.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.