Reproduce “From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Models” on OpenResearch

About this paper

Sean Welleck, Amanda Bertsch, Matthew Finlayson, Hailey Schoelkopf, Alex Xie, Graham Neubig, Ilia Kulikov, Zaid Harchaoui

A comprehensive survey from researchers at Carnegie Mellon University, EleutherAI, Meta FAIR, University of Southern California, and University of Washington unifies inference-time algorithms for Large Language Models into a coherent framework. It categorizes techniques across token-level generation, meta-generation, and efficiency, demonstrating how scaling compute during inference significantly improves LLM outputs and addresses inherent model limitations, complementing advances in training-time scaling.

Key insights

• Scaling compute during inference, especially through meta-generation algorithms, substantially improves LLM outputs for complex tasks by enabling dynamic computation.
• The integration of external information and programmatic control (e.g., chained, parallel, search, refinement patterns) is crucial for augmenting LLM capabilities and addressing inherent model limitations.
• Efficient generation requires balancing output quality with computational cost, emphasizing 'compute-optimal inference' and specific optimizations like KV Cache reuse for multi-call strategies.

Summary by alphaXiv.