Reproduce “DeepSeek vs. ChatGPT vs. Claude: A Comparative Study for Scientific Computing and Scientific Machine Learning Tasks” on OpenResearch

About this paper

Qile Jiang, Zhiwei Gao, George Em Karniadakis

Researchers from Brown University conducted a comprehensive comparative study of advanced Large Language Models, including their reasoning-optimized variants, on challenging scientific computing and scientific machine learning tasks like stiff ODEs and DeepONets. The study found that reasoning-optimized models generally outperformed general-purpose LLMs in problem analysis and method selection, although all models frequently generated code containing bugs or inefficiencies that necessitated human debugging.

Key insights

• Reasoning-optimized LLMs consistently outperform general-purpose models, demonstrating superior capabilities in problem analysis and selecting appropriate advanced scientific methods.
• Even state-of-the-art LLMs frequently generate code with bugs or inefficiencies, particularly in complex scientific machine learning tasks, necessitating significant human debugging and expert oversight.
• LLMs exhibit limitations in handling ambiguous prompts and making crucial contextual decisions in scientific problems, such as defining input function spaces for DeepONets or handling arbitrary fractional orders.

Summary by alphaXiv.