Reproduce “Efficient multi-prompt evaluation of LLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Felipe Maia Polo, Ronald Xu, Lucas Weber, Mírian Silva, Onkar Bhardwaj, Leshem Choshen, Allysson Flavio Melo de Oliveira, Yuekai Sun and 1 more
PromptEval, developed by researchers including those from the MIT-IBM Watson AI Lab, introduces an efficient method based on Item Response Theory to estimate the full performance distribution of Large Language Models across a large pool of prompt templates. It provides accurate performance quantiles with a budget equivalent to just two single-prompt evaluations, addressing the high computational cost and prompt sensitivity of current LLM evaluation practices.
Key insights
- PromptEval significantly reduces the computational cost of comprehensive multi-prompt evaluation while maintaining high accuracy.
- LLM performance is highly sensitive to prompt variations not only overall but also at the individual subject level, with low cross-model consistency for effective prompts.
- Estimating performance distributions and quantiles offers a more robust and nuanced understanding of LLM capabilities than single-point metrics.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.