Reproduce “Done Is Not Correct: Measuring Silent Failures and Self-Verification Calibration When LLM Agents Take CAD Actions” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Carson Rodrigues, Clive Rodrigues, Aravind Reddy G
The authors introduce a CAD benchmark testing whether agents’ completion claims match measured geometry. Silent failure requires a valid, renderable solid, an explicit completion claim, and a failed semantic check; crashes and non-claims are excluded. Across four models and 13 tasks, measure-then-claim tool-use reduced silent failures from 22/156 (14.1%) to 5/156 (3.2%), a result bounded to this suite, not proof of manufacturability or safety.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.