Reproduce “TheoremQA: A Theorem-driven Question Answering dataset” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Wenhu Chen, Ming Yin, Max Ku, Pan Lu, Yixin Wan, Xueguang Ma, Jianyu Xu, Xinyi Wang and 1 more
TheoremQA introduces the first theorem-driven question-answering dataset, designed to assess large language models' ability to apply domain-specific scientific theorems. The benchmark, covering university-level problems in Math, Physics, EE&CS, and Finance, reveals GPT-4 achieves 51% accuracy, significantly outperforming other models.
Key insights
- Program-of-Thought (PoT) prompting significantly boosts performance for GPT-series models by offloading complex computations to an external executor, with GPT-4 achieving a 92% program executability rate.
- Current LLMs, including multimodal instruction-tuned models, struggle severely with interpreting complex scientific diagrams, often performing below 10% accuracy on visual questions.
- Simply augmenting prompts with explicit theorem descriptions yields only marginal improvements, suggesting LLMs need more sophisticated methods for integrating and applying abstract knowledge.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.