Reproduce “Prompt Optimization with EASE? Efficient Ordering-aware Automated Selection of Exemplars” on OpenResearch

About this paper

Zhaoxuan Wu, Xiaoqiang Lin, Zhongxiang Dai, Wenyang Hu, Yao Shu, See-Kiong Ng, Patrick Jaillet, Bryan Kian Hsiang Low

Researchers from the National University of Singapore and collaborators developed EASE, an automated method for selecting fixed, ordered in-context exemplars for black-box Large Language Models, which also enables joint optimization of instructions and exemplars. The method achieved superior accuracy in 17 out of 19 language tasks and an 8.3% absolute accuracy gain over competitive baselines in novel rule-based tasks, significantly enhancing in-context learning.

Key insights

• Exemplar selection's impact on LLM performance is more pronounced when the model has limited prior knowledge about a task, with performance gains diminishing as models are increasingly fine-tuned.
• Explicitly accounting for exemplar ordering within prompts, rather than just subset selection, significantly improves in-context learning performance.
• Joint optimization of both instructional prompts and in-context exemplars yields synergistic benefits, leading to higher overall prompt performance than optimizing them independently.

Summary by alphaXiv.