Reproduce “RTL-Repo: A Benchmark for Evaluating LLMs on Large-Scale RTL Design Projects” on OpenResearch

About this paper

Ahmed Allam, Mohamed Shalan

The paper introduces RTL-Repo, a benchmark for evaluating Large Language Models (LLMs) on multi-file Register Transfer Level (RTL) design projects. It demonstrates that advanced LLMs like GPT-4 achieve an Edit Similarity of 71.87 and an Exact Match of 48.5, but all tested models exhibit a substantial performance decline as context length increases.

Key insights

• Current LLMs, including those with large context windows, demonstrate significant performance degradation when faced with long-range dependencies and extensive multi-file contexts in Verilog.
• Larger, general-purpose LLMs (e.g., GPT-4) outperform smaller, Verilog-specific open-source models on complex, multi-file RTL tasks, indicating that current domain-specific fine-tuning on smaller models is insufficient for these challenges.
• RTL-Repo's automated construction as a 'living benchmark' helps mitigate data leakage issues and ensures its long-term relevance for guiding the development of more capable LLMs in hardware design.

Summary by alphaXiv.