Reproduce “Nudging: Inference-time Alignment of LLMs via Guided Decoding” on OpenResearch

About this paper

Yu Fei, Yasaman Razeghi, Sameer Singh

NUDGING introduces a training-free, inference-time method for aligning Large Language Models by leveraging a smaller, already aligned model to guide a larger base model at the token level. This approach achieves performance comparable to fully fine-tuned models, often surpassing them, while substantially reducing alignment computational costs and demonstrating effective cross-family model collaboration.

Key insights

• LLM alignment primarily refines a small subset of "stylistic tokens" (e.g., discourse markers, formatting), and base models are highly uncertain when generating these alignment-related tokens.
• Smaller aligned models produce token distributions similar to large aligned models at these critical alignment-related positions, indicating their efficacy as "nudging" sources.
• Inference-time, token-level collaboration between an unaligned base model and a smaller aligned model offers a highly efficient and effective alternative to costly full model re-training for alignment, achieving comparable or superior results.

Summary by alphaXiv.