Reproduce “Analogy Generation by Prompting Large Language Models: A Case Study of InstructGPT” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Bhavya Bhavya, Jinjun Xiong, Chengxiang Zhai
Researchers from the University of Illinois at Urbana-Champaign and the University at Buffalo systematically studied the application of prompting InstructGPT for generating complex textual analogies, demonstrating its feasibility and revealing sensitivities to prompt design and model size. The largest InstructGPT model achieved 70.05% meaningful analogies when generating new analogous concepts, comparable to human performance of 66.67%.
Key insights
- InstructGPT's effectiveness in generating analogies is highly sensitive to prompt design, with imperative statements significantly outperforming questions.
- Lower temperature settings generally yield higher quality analogies compared to high temperatures, suggesting a preference for less randomness for accuracy against reference data.
- Analogy generation quality increases robustly with model size, with the largest InstructGPT (Davinci) showing capabilities comparable to human performance for generating analogous concepts.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.