Reproduce “RealBench: Benchmarking Verilog Generation Models with Real-World IP Designs” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Pengwei Jin, Di Huang, Chongxiao Li, Shuyao Cheng, Yang Zhao, Xinyao Zheng, Jiaguo Zhu, Shuyi Xing and 5 more

Researchers introduce RealBench, a benchmark for evaluating Verilog generation models that uses real-world IP designs, multi-modal specifications, and rigorous formal verification. Evaluations demonstrate that current state-of-the-art large language models exhibit low accuracy on complex hardware design tasks, particularly for hierarchical designs and submodule instantiations.

Key insights

  • Current LLMs show a profound performance gap, achieving only 13.3% formal accuracy on module-level tasks and 0% on system-level tasks in real-world IP generation, indicating a severe limitation for practical application.
  • Formal verification is indispensable for accurate assessment, as a substantial portion (44.2% on RTLLMV2) of code passing testbenches fails formal equivalence checking, overestimating LLM accuracy.
  • Multi-modal LLMs struggle to integrate complex visual information from diverse diagrams within lengthy specifications, showing only marginal improvements over text-only models.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.