Reproduce “PICBench: Benchmarking LLMs for Photonic Integrated Circuits Design” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yuchao Wu, Xiaofei Yu, Hao Chen, Yang Luo, Yeyu Tong, Yuzhe Ma

HKUST(GZ) researchers developed PICBench, an open-source framework for benchmarking Large Language Models (LLMs) in generating simulation-ready netlists for Photonic Integrated Circuits. The framework demonstrates that incorporating iterative error feedback and domain-specific restrictions substantially improves LLM performance, with some models achieving near-perfect syntax correctness (up to 100% Pass@5) when guided by restrictions and multiple feedback iterations.

Key insights

  • Iterative error feedback from simulator diagnostics significantly improves LLM performance in generating both syntactically and functionally correct PIC netlists, accelerating the debugging process.
  • Explicit, domain-specific restrictions integrated into the system prompt dramatically enhance LLMs' ability to adhere to specialized design rules, leading to substantial gains in syntax correctness.
  • While LLMs can achieve high syntax accuracy with proper guidance, mastering the intricate functional logic of complex PIC designs presents a more significant challenge compared to learning specific syntax rules.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.