Reproduce “ShapeBench: A Scalable Benchmark and Diagnostic Suite for Standardized Evaluation in Aerodynamic Shape Optimization” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shaghayegh Fazliani, Krissh Chawla, Jack Guo, Yiren Shen, Matthias Ihme, Madeleine Udell
ShapeBench is introduced as an open-source, scalable benchmark and diagnostic suite for aerodynamic shape optimization (ASO). It provides a standardized framework for evaluating classical and LLM-driven optimization methods across diverse aerodynamic tasks, revealing that optimizer performance does not generalize reliably and identifying surrogate exploitation issues through its integrated diagnostic tools.
Key insights
- Optimizer performance is highly variable across different aerodynamic problem classes, with a mean pairwise Spearman \u03c1 of approximately 0.013 between environment-level rankings, indicating poor generalization from single-task studies.
- Surrogate models are susceptible to exploitation by optimizers, leading to high objective values for physically implausible or infeasible designs, highlighting the necessity for robust diagnostic checks.
- Different objective formulations can lead to local optima traps, as observed in the 3D Blended Wing Body (BWB) multi-point tasks, where methods converged to suboptimal designs depending on initialization.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.