Reproduce “NetlistBench: Evaluating LLM Reliability in SPICE Netlist Recognition and Manipulation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jiarui Ma, Jianghan Wang, Yuheng Ma, Ziyi Zhuang, Xiaoguang Liu
NetlistBench evaluates whether models can recognize and edit circuit netlists while preserving their structure, using 2,342 cases across 24 task families and canonical circuit representations for grading. Local edits reach 96%–100%, but dependent compound-edit performance declines with horizon: Claude Sonnet 4.6 falls from 80% at 3 edits to 26% at 15, within bounded analog CMOS blocks.
Key insights
- LLMs are generally reliable for localized SPICE netlist edits and explicit attribute extraction but exhibit significantly lower reliability for operations requiring the maintenance or introduction of complex structural connectivity or hierarchy.
- Performance sharply degrades as the number of dependent edits in compound tasks increases, indicating that reliability on isolated operations does not translate to robust multi-step transformations.
- While reasoning modes (native thinking, Chain-of-Thought) can provide substantial gains (30-40 percentage points) in aggregate performance, they do not fully resolve issues with long-horizon compound edits, hierarchical operations, or relational structural queries.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.