Reproduce “Survival of the Most Influential Prompts: Efficient Black-Box Prompt Search via Clustering and Pruning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Han Zhou, Xingchen Wan, Ivan Vulić, Anna Korhonen

A method for optimizing black-box hard prompt search, CLAPS, dramatically reduces the search space for large language models by identifying and pruning non-influential tokens. This approach, developed by researchers at the University of Cambridge and Oxford, yields up to 1.8% higher average accuracy on GLUE and XNLI tasks compared to prior state-of-the-art methods while simultaneously reducing model query costs by as much as 97%.

Key insights

  • A vast majority (over 99%) of tokens within an LLM's vocabulary are either non-influential or detrimental when used as discrete prompts.
  • Optimizing the search space for discrete prompts is more critical for improving black-box HPT efficiency and performance than solely focusing on developing complex search algorithms.
  • Even simple search strategies, such as greedy search, can achieve competitive performance when applied to an intelligently pruned and optimized prompt search space.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.