Reproduce “Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, Pontus Stenetorp

Researchers from University College London identify that the performance of few-shot in-context learning with large language models is highly sensitive to the order of examples within the prompt. They introduce an entropy-based probing method that generates an artificial, unlabeled probing set to select performant prompt orderings, achieving an average of 13% relative performance improvement and reduced variance across various text classification tasks without requiring additional labeled data.

Key insights

  • The order of few-shot examples within a prompt is a critical, previously underexplored factor that causes dramatic performance fluctuations in in-context learning.
  • Good prompt orderings are not transferable across different models or tasks, and poor orders often lead to degenerate model behavior, such as consistently biasing towards a single label.
  • It is possible to effectively identify performant prompt orderings by leveraging the language model's generative power to create an artificial, unlabeled development set.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.