Reproduce “Calibrate Before Use: Improving Few-Shot Performance of Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tony Z. Zhao, Eric Wallace, Shi Feng, Dan Klein, Sameer Singh
Contextual calibration, a data-free method, significantly improves the few-shot performance and stability of large language models like GPT-3 and GPT-2 by mitigating their inherent contextual biases. This approach demonstrably reduces accuracy variance and allows smaller calibrated models to outperform larger uncalibrated ones.
Key insights
- Large language models exhibit specific inherent biases (majority label, recency, common token bias) that cause their few-shot performance to be highly unstable.
- These biases lead to a systematic shift in the model's output probability distribution, favoring certain answers irrespective of the input content.
- These contextual biases can be estimated and mitigated effectively by analyzing the model's predictions on content-free inputs within the few-shot prompt.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.