Reproduce “Efficient and Accurate Prompt Optimization: the Benefit of Memory in Exemplar-Guided Reflection” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Cilin Yan, Jingyun Wang, Lin Zhang, Ruihui Zhao, Xiaopu Wu, Kai Xiong, Qingsong Liu, Guoliang Kang and 1 more
Exemplar-Guided Reflection with Memory (ERM) is a method for automated prompt optimization for large language models that integrates memory mechanisms to store and reuse historical feedbacks and exemplars. This approach achieves improved accuracy across seven diverse datasets while significantly reducing the number of optimization steps and total token consumption.
Key insights
- Explicitly managing and leveraging historical feedback and exemplars through memory mechanisms can drastically improve the efficiency and accuracy of automated prompt optimization.
- Generating detailed solution processes (Chain-of-Thought) for erroneous examples leads to more informative feedback and higher quality exemplars for in-context learning.
- Filtering and selectively forgetting less valuable feedbacks and exemplars are crucial for maintaining memory quality and preventing the accumulation of irrelevant information.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.