Reproduce “CITE: Anytime-Valid Statistical Inference in LLM Self-Consistency” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hirofumi Ota, Naoto Iwase, Yuki Ichihara, Junpei Komiyama, Masaaki Imaizumi
The CITE algorithm offers an anytime-valid statistical inference method for LLM self-consistency, provably controlling the Type-I error rate of falsely certifying a unique modal response, even with unknown answer categories. This approach demonstrates improved efficiency and reliability compared to existing baselines across simulated and LLM benchmarks.
Key insights
- Anytime-valid statistical inference, leveraging martingale theory and e-processes, allows for rigorous Type-I error control uniformly over all possible data-dependent stopping times.
- The problem of certifying a unique mode with an unknown and expanding category set can be robustly addressed by simultaneously monitoring evidence against observed, hidden, and entirely unseen competitors.
- CITE's stopping time demonstrates minimax optimality, achieving an efficient rate that is independent of the total number of possible response categories.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.