Reproduce “Conformal Prediction with Large Language Models for Multi-Choice Question Answering” on OpenResearch

About this paper

Bhawesh Kumar, Charlie Lu, Gauri Gupta, Anil Palepu, David Bellamy, Ramesh Raskar, Andrew Beam

Harvard University and MIT Media Lab researchers applied conformal prediction to the LLaMA-13B model for multi-choice question answering, establishing a method to provide formal, statistically guaranteed error rates for LLM predictions. This approach consistently achieved a target coverage (e.g., 90%) across various subjects and enabled selective classification by correlating prediction set size with accuracy.

Key insights

• Conformal prediction successfully provides statistical guarantees on error rates for LLM predictions in MCQA tasks, ensuring that prediction sets contain the true label with a specified probability.
• The size of the prediction set generated by conformal prediction reliably indicates the model's uncertainty, showing a robust negative correlation with the top-1 accuracy of predictions.
• While the exchangeability assumption is crucial for strict coverage guarantees, deviations were minor when calibrating and evaluating on related subject domains, suggesting some generalizability for LLM softmax probabilities.

Summary by alphaXiv.