Reproduce “A Survey of Large Language Models” on OpenResearch

About this paper

Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang and 14 more

The authors survey large language models across pre-training, adaptation, utilization, and evaluation, organizing techniques, evidence, and resources across the model lifecycle. They describe a shift toward prompt-based use while noting unresolved principles, compute demands, and evaluation limits. After ChatGPT’s release, daily arXiv papers matching the exact phrase rose from 0.40 to 8.58. This measures publication activity, not capability growth.

Key insights

• Large Language Models exhibit non-linear 'emergent abilities' like in-context learning and Chain-of-Thought reasoning upon reaching certain scales, fundamentally changing their problem-solving capabilities.
• The quality and scale of pre-training data, alongside optimized architectures (predominantly decoder-only Transformers) and advanced distributed training techniques, are paramount for developing capable LLMs.
• Post-training adaptation, including instruction tuning and alignment tuning (e.g., RLHF), is critical for enhancing LLMs' ability to follow instructions, generalize, and align with human values for safe and effective deployment.

Summary by alphaXiv.