Reproduce “Instance-adaptive Zero-shot Chain-of-Thought Prompting” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Xiaosong Yuan, Chen Shen, Shaotian Yan, Xiaofeng Zhang, Liang Xie, Wenxiao Wang, Renchu Guan, Ying Wang and 1 more

Researchers at Jilin University, Alibaba Cloud Computing, and Shanghai Jiao Tong University devised an instance-adaptive prompting strategy to enhance zero-shot Chain-of-Thought reasoning, showing how tailoring prompts based on internal information flow analysis significantly improves LLM accuracy across mathematical, logical, and commonsense tasks by 2-4%.

Key insights

  • Successful zero-shot CoT reasoning is characterized by strong semantic information transfer, where the prompt effectively gathers question information (high Iqp) in shallow layers, and the rationale aggregates information from both the question (Iqr) and the question-informed prompt (Ipr).
  • Conversely, failed reasoning instances show significantly lower saliency scores in these critical information flow paths, indicating a breakdown in prompt effectiveness or rationale integration.
  • Specific Transformer layers (shallow ones for Iqp) and attention heads play specialized roles in mediating these crucial information flows, contributing to the overall reasoning process.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.