Reproduce “MaScQA: A Question Answering Dataset for Investigating Materials Science Knowledge of Large Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Mohd Zaki, Jayadeva, Mausam, N. M. Anoop Krishnan

Researchers at the Indian Institute of Technology Delhi introduced MaScQA, a question-answering dataset of 650 questions derived from the GATE examination, to benchmark large language models' understanding of materials science. GPT-4 achieved an overall accuracy of 62.0% on this dataset, demonstrating strong performance on matching tasks but revealing significant limitations in numerical reasoning and a high prevalence of conceptual errors (64%).

Key insights

  • GPT-4 significantly outperforms GPT-3.5 in materials science question answering, but both models exhibit notable limitations in conceptual understanding and numerical reasoning within the domain.
  • Chain of Thought prompting does not consistently improve LLM performance in materials science and can even decrease accuracy for specific numerical question types, contrasting its general benefits in other domains.
  • Conceptual errors (64% of mistakes) are the primary limiting factor for LLMs in materials science, followed by computational errors (36%), while CoT prompting largely eliminated grounding errors.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.