Reproduce “The Prompt Report: A Systematic Survey of Prompt Engineering Techniques” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Sander Schulhoff, Michael Ilie, Nishant Balepur, Konstantine Kahadze, Amanda Liu, Chenglei Si, Yinheng Li, Aayush Gupta and 23 more

Researchers from the University of Maryland, OpenAI, Stanford University, and other institutions present a systematic survey of prompt engineering, developing a comprehensive taxonomy of 58 LLM prompting techniques and a standardized vocabulary. Their work includes empirical benchmarking on MMLU, which generally shows performance improvements with more complex prompting, and a case study demonstrating that automated prompt optimization can outperform human-designed prompts.

Key insights

  • Manual prompt engineering is an iterative, complex, and highly sensitive process where subtle changes can significantly impact performance, requiring deep understanding and domain expertise.
  • Automated prompt optimization frameworks like DSPy show promise, demonstrating the potential to outperform human expert-designed prompts.
  • A standardized vocabulary and comprehensive taxonomy are crucial for advancing research and practical application in the rapidly evolving field of prompt engineering.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.