Reproduce “Convex Optimization for Alignment and Preference Learning on a Single GPU” on OpenResearch

About this paper

Miria Feng, Mert Pilanci

COALA, a method for Large Language Model (LLM) preference alignment, leverages convex optimization to enable efficient and stable fine-tuning on a single GPU. It achieves competitive human preference win rates while significantly reducing computational demands, using approximately 17.6% of the TFLOPs typically required by DPO.

Key insights

• Convex optimization, specifically for a small network head atop a frozen LLM, offers a theoretically robust and computationally efficient alternative for preference alignment.
• Significant computational savings and training stability can be achieved by reformulating a portion of the alignment problem as a convex program, enabling single-GPU training for models like Llama-3.1-8B.
• The "Alternating Population Strategy" provides a sample-efficient method to generate preference datasets from multi-turn conversations, bypassing the need for external LLM re-ranking.

Summary by alphaXiv.