Reproduce “ChipNeMo: Domain-Adapted LLMs for Chip Design” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Mingjie Liu, Teodor-Dumitru Ene, Robert Kirby, Chris Cheng, Nathaniel Pinckney, Rongjian Liang, Jonah Alben, Himyanshu Anand and 34 more

ChipNeMo adapts LLaMA 2 to chip-design code and documents through domain-adaptive pretraining, instruction alignment, and retrieval for document-specific questions. On an internal Python EDA-tool benchmark, ChipNeMo-70B-Steer achieved 49% Pass@5 on Easy tasks versus GPT-4’s 0%. With retrieval, it averaged 5.26 versus 4.52 for chatbot responses, but GPT-4 led on bug evaluation.

Key insights

  • Comprehensive domain adaptation, particularly Domain-Adaptive Pretraining (DAPT), is highly effective and cost-efficient for specializing LLMs, costing less than 1.5% of training a foundation model from scratch.
  • Employing a low learning rate during DAPT is crucial for distilling domain knowledge while preserving general language capabilities, preventing significant degradation on broader tasks.
  • Advanced alignment methods like SteerLM and a custom, domain-adapted RAG system significantly improve human evaluation scores and retrieval accuracy for specialized chatbot applications.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.