Reproduce “ChipVerilog: A Large-Scale OpenCores-Derived Benchmark for LLM-Based Verilog RTL Generation” on OpenResearch

About this paper

Yan Tan, Jiping Du, Xiangchen Meng, Yangdi Lyu

ChipVerilog introduces a 64-target Verilog RTL generation benchmark built from five OpenCores-derived design families, totaling roughly 20,400 lines and spanning standalone modules, hierarchical components, and full systems. Evaluation of Claude Opus 4.5, GPT-5.4, and DeepSeek V4 Pro shows that syntax generation is substantially easier than functional reproduction: GPT-5.4 reaches 83.13% syntax pass@1 but only 23.13% functional pass@1, with performance dropping sharply for RTL longer than 300 lines or involving multiple submodules.

Key insights

• Compilability and behavioral correctness are widely separated: repeated sampling raises syntax pass@5 substantially, but produces much smaller gains in functional correctness.
• Long RTL creates a semantic and structural bottleneck rather than merely a parsing challenge; functional pass@1 falls from about 58.0% below 100 lines to 3.2% for 300–500-line targets and 0% above 500 lines.
• Multiple submodules are especially difficult: designs with two or more submodules have 55.69% syntax pass@1 but 0% functional pass@1 and pass@5, indicating problems with wiring, dependencies, timing, and cross-module coordination.

Summary by alphaXiv.