Reproduce “Benchmarking PhD-Level Coding in 3D Geometric Computer Vision” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Wenyi Li, Renkai Luo, Yue Yu, Huan-ang Gao, Mingju Gao, Li Yuan, Chaoyou Fu, Hao Zhao
GeoCodeBench, a new benchmark, assesses large language models' (LLMs) ability to perform "PhD-level" coding in 3D geometric computer vision, revealing that leading models achieve only modest pass rates (e.g., GPT-5 at 36.6%) and struggle with complex scientific text comprehension. The findings indicate a significant gap in LLMs' capability to translate intricate research paper content into correct, executable code for specialized scientific applications.
Key insights
- State-of-the-art LLMs, including GPT-5, exhibit substantial limitations, achieving a best pass rate of 36.6% on PhD-level 3D geometric vision coding, indicating a significant reliability gap.
- LLMs demonstrate relatively stronger foundational 3D understanding but struggle with higher-level procedural reasoning and algorithmic synthesis required for novel research implementations.
- Providing full paper context does not consistently improve performance; instead, truncated, method-focused inputs often yield better results, suggesting challenges in long-context scientific comprehension and noise management for LLMs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.