Reproduce “NotSoTiny: A Large, Living Benchmark for RTL Code Generation” on OpenResearch

About this paper

Razine Moundir Ghorab, Emanuele Parisi, Cristian Gutierrez, Miquel Alberti-Binimelis, Miquel Moreto, Dario Garcia-Gasulla, Gokcen Kestor

A large, continually updated benchmark for Register-Transfer Level (RTL) code generation, NotSoTiny, employs formal verification and real-world hardware designs to rigorously evaluate Large Language Models. It exposes a substantial gap, with leading LLMs achieving only 20-23% functional correctness, despite high syntactic accuracy, while demonstrating the inadequacy of traditional simulation testbenches.

Key insights

• LLM performance on RTL generation is significantly overestimated by traditional simulation metrics; formal verification reveals a substantial gap between syntactic validity and functional correctness.
• Effective benchmarks for LLMs in hardware design require higher complexity, hierarchical structure, and context-aware tasks to reflect real-world engineering challenges and drive meaningful model improvements.
• Continuous mitigation of data contamination is crucial for valid LLM evaluation in code generation, emphasizing the need for dynamic and periodically updated benchmarks.

Summary by alphaXiv.