Reproduce “ArchXBench: A Complex Digital Systems Benchmark Suite for LLM Driven RTL Synthesis” on OpenResearch

About this paper

Suresh Purini, Siddhant Garg, Mudit Gaur, Sankalp Bhat, Sohan Mupparapu, Arun Ravindran

ArchXBench, a new benchmark suite, is introduced to evaluate large language models (LLMs) on the synthesis of complex digital systems at the Register-Transfer-Level (RTL). Initial evaluations using this suite identified a "complexity cliff" where LLMs struggle with intricate designs beyond basic levels, although models like o4-mini-high demonstrated stronger performance on foundational benchmarks and Sonnet 4.0 notably succeeded on a complete AES encryption core.

Key insights

• Current state-of-the-art LLMs encounter a "complexity cliff," showing a sharp decline in performance for generating complex RTL designs beyond simpler levels.
• LLM success in RTL synthesis appears influenced by the specificity of mathematical descriptions and familiarity with certain designs (e.g., AES) within their training data.
• The inability of LLMs to consistently produce correct and architecturally compliant code for complex, bespoke designs indicates a need for enhanced architectural reasoning and multi-step planning capabilities.

Summary by alphaXiv.