Reproduce “Rule2DRC: Benchmarking LLM Agents for DRC Script Synthesis with Execution-Guided Test Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jinuk Kim, Junsoo Byun, Donghwi Hwang, Seong-Jin Park, Hyun Oh Song
This research introduces Rule2DRC, a large-scale, open-source benchmark for translating natural language design rules into executable Design Rule Checking (DRC) scripts, and proposes SplitTester, an LLM-based agent that improves script selection by systematically generating execution-guided, discriminative test cases. SplitTester consistently achieves higher success rates and lower error rates compared to baseline methods, effectively bridging the gap between LLMs and functional correctness in chip design automation.
Key insights
- Providing comprehensive domain-specific API documentation (e.g., 60K tokens of KLayout API) significantly boosts LLM performance for specialized code generation tasks.
- Systematically generating discriminative test cases by targeting and splitting clusters of indistinguishable candidate programs is more effective for selecting functionally correct code than static or less targeted testing strategies.
- A robust, open-source benchmark with execution-based functional correctness evaluation is essential for advancing and rigorously comparing LLM agents in complex engineering domains like Electronic Design Automation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.