Reproduce “A Thorough Examination of Decoding Methods in the Era of LLMs” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Chufan Shi, Haoran Yang, Deng Cai, Zhisong Zhang, Yifan Wang, Yujiu Yang, Wai Lam

A comprehensive empirical study evaluated 13 decoding methods across 11 NLP tasks and various Large Language Models, establishing that no single method is universally optimal and showing that model alignment significantly reduces the performance variance attributable to decoding choices. The research also offers actionable guidelines for method selection based on task and deployment context.

Key insights

  • There is no universally optimal decoding method; the best choice is highly dependent on the specific task, the LLM used (aligned vs. unaligned), and the priority (e.g., performance, robustness, or speed).
  • Model alignment (e.g., Llama2-Chat) significantly reduces the performance variation across different decoding methods due to improved model confidence and alleviated degeneration, making decoding method choice less critical for aligned models.
  • Newer deterministic methods like FSD and FSD-d demonstrate both high performance and robustness to hyperparameter changes, making them practical choices for real-world applications where extensive tuning is not feasible.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.