Reproduce “Cautious Next Token Prediction” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yizhou Wang, Lingzhi Zhang, Yue Bai, Mang Tik Chiu, Zhengmian Hu, Mingyuan Zhang, Qihua Dong, Yu Yin and 2 more
Cautious Next Token Prediction (CNTP) is an adaptive, training-free decoding strategy that dynamically adjusts the number of sampled continuations based on the language model's prediction entropy. This approach improves accuracy and truthfulness in LLMs and MLLMs, achieving 81.6% accuracy on GSM8K with Llama-3.1-8B-Instruct and boosting truthfulness to 84.8% on Truthful-QA.
Key insights
- Dynamically scaling the depth of local path exploration based on the model's internal uncertainty (prediction entropy) allows for efficient resource allocation, engaging in deeper search only when needed.
- Evaluating and selecting local continuations based on their perplexity provides an intrinsic, model-driven measure of coherence and likelihood, leading to more reliable outputs.
- This adaptive, uncertainty-aware decoding strategy effectively bridges the gap between fast, single-sample methods and robust, multi-sample reasoning, achieving performance gains with improved efficiency.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.