Reproduce “In-context Example Selection with Influences” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tai Nguyen, Eric Wong
Researchers from the University of Pennsylvania introduce 'in-context influences,' a method adapting retraining-based influence functions to quantify the impact of individual in-context examples on Large Language Model performance. This approach enables the selection of highly beneficial examples, outperforming existing heuristics, and reveals a distinct signal for example utility in in-context learning.
Key insights
- In-context influences capture a distinct signal of example utility, showing no correlation (R^2 = 0.0) with common heuristics like perplexity, input length, or semantic similarity.
- The method is computationally efficient for LLMs, requiring only multiple forward passes instead of costly model retraining.
- Example influences can be used to systematically analyze emergent LLM phenomena, such as confirming and quantifying recency bias in example ordering.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.