Reproduce “Instances Need More Care: Rewriting Prompts for Instances with LLMs in the Loop Yields Better Zero-Shot Performance” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Saurabh Srivastava, Chengyue Huang, Weiguo Fan, Ziyu Yao
PROMPTED, a method developed by researchers at George Mason University and the University of Iowa, enhances zero-shot Large Language Model performance by iteratively rewriting input prompts at the individual instance level using an LLM in the loop. The approach achieved an average performance gain of at least 6% over strong baselines and demonstrated up to 50% absolute improvement on tasks requiring logical reasoning, also showing that weaker LLMs can effectively supervise stronger ones.
Key insights
- Iterative, instance-level input prompt optimization with an LLM in the loop is an effective strategy for enhancing zero-shot performance.
- Rewriting the input prompt can introduce new reasoning paths and address fundamental task misunderstandings more effectively than merely refining the output.
- Weaker LLMs can successfully supervise and improve the performance of stronger LLMs through instance-level prompt generation, offering cost-efficiency and new architectural possibilities.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.