Reproduce “From Medprompt to o1: Exploration of Run-Time Strategies for Medical Challenge Problems and Beyond” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Harsha Nori, Naoto Usuyama, Nicholas King, Scott Mayer McKinney, Xavier Fernandes, Sheng Zhang, Eric Horvitz
Microsoft and OpenAI investigated the impact of traditional prompt engineering strategies on OpenAI's 'reasoning-native' o1-preview model, particularly in medical contexts. The research found that o1-preview achieves 96.0% accuracy on MedQA with minimal prompting, and that many traditional external prompting techniques offer diminishing returns or can even hinder performance, while ensemble methods continue to provide consistent improvements.
Key insights
- Reasoning-native LLMs internalize complex reasoning, leading to diminishing returns or even performance degradation from traditional prompt engineering techniques like few-shot examples or detailed instructions.
- Ensemble methods continue to provide consistent performance improvements by reducing variance, even for advanced models that internally perform reasoning.
- Encouraging more extensive internal reasoning (e.g., 'Extended Reasoning' vs. 'Quick Response' prompts) in models like o1-preview generally correlates with improved accuracy, highlighting the importance of inference-time computation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.