Reproduce “Prompting Science Report 1: Prompt Engineering is Complicated and Contingent” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Lennart Meincke, Ethan Mollick, Lilach Mollick, Dan Shapiro
Researchers from Generative AI Labs at The Wharton School demonstrated that Large Language Model performance, evaluated on a challenging PhD-level question set, is highly contingent on the consistency thresholds used and the specificity of prompt instructions. Their analysis revealed that models like GPT-4o often do not significantly outperform random chance under 100% consistency requirements, and that explicit formatting instructions consistently enhance reliability, while "politeness" in prompts shows highly variable, context-dependent effects.
Key insights
- LLM performance and reliability are highly contingent on the chosen evaluation standard, with model capabilities appearing substantially different under varying consistency thresholds.
- Explicit formatting instructions in prompts consistently improve LLM performance and consistency, underscoring the importance of clear structural guidance.
- The impact of conversational prompt elements like "politeness" is not universally beneficial; its effectiveness is highly contingent and varies significantly across individual questions rather than showing an aggregated effect.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.