Reproduce “Batch Calibration: Rethinking Calibration for In-Context Learning and Prompt Engineering” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Han Zhou, Xingchen Wan, Lev Proleev, Diana Mincu, Jilin Chen, Katherine Heller, Subhrajit Roy
Batch Calibration (BC) reduces prompt-dependent classification bias by estimating each class’s average score across the unlabeled batch, rather than using a dummy example. It subtracts these estimates from individual scores before prediction. In 1-shot PaLM 2-S experiments across 13 natural-language tasks, BC averaged 74.41% accuracy, versus 67.65% for Prototypical Calibration. This aggregate does not mean BC leads on every task.
Key insights
- Content-free tokens, commonly used by prior methods, can introduce additional bias, leading to inaccurate prior estimations and calibration failures, especially in multi-sentence classification tasks.
- Estimating contextual bias directly from content-based batched inputs provides a more robust and stable linear decision boundary compared to complex, often unstable non-linear boundaries.
- The principle of contextual bias and its mitigation through batch-wise marginalization is broadly applicable across AI modalities, effectively addressing biases in both LLMs and Vision-Language Models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.