Reproduce “API Is Enough: Conformal Prediction for Large Language Models Without Logit-Access” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jiayuan Su, Jing Luo, Hongwei Wang, Lu Cheng
LofreeCP introduces a conformal prediction method for large language models that quantifies uncertainty without requiring access to internal logits. It achieves superior prediction set efficiency and maintains statistical coverage guarantees by using a combination of response frequency, normalized entropy, and semantic similarity, outperforming logit-based and other logit-free baselines across QA and MCQ tasks.
Key insights
- Empirical analysis shows a strong correlation between response frequency and true output probability, making frequency an effective proxy for ranking response uncertainty.
- Combining coarse-grained response frequency with fine-grained normalized entropy and semantic similarity effectively mitigates the "concentration issue" of similar nonconformity scores, leading to more efficient prediction sets.
- LofreeCP maintains the rigorous statistical coverage guarantee inherent to Conformal Prediction by satisfying the nesting property.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.