Reproduce “Dynamic Rewarding with Prompt Optimization Enables Tuning-free Self-Alignment of Language Models” on OpenResearch

About this paper

Somanshu Singla, Zhen Wang, Tianyang Liu, Abdullah Ashfaq, Zhiting Hu, Eric P. Xing

This paper introduces Dynamic Rewarding with Prompt Optimization (DRPO), a tuning-free approach that self-aligns Large Language Models by leveraging a search-based optimization framework and a dynamic reward mechanism. Empirical evaluations across eight diverse LLMs show DRPO enables base models to often surpass their SFT or RLHF-tuned counterparts in alignment performance.

Key insights

• LLMs possess significant inherent alignment capabilities that can be activated through sophisticated inference-time prompt optimization without costly fine-tuning.
• Dynamically tailoring reward criteria based on specific queries allows for the identification and correction of model-specific alignment weaknesses.
• Framing LLM alignment as a search-based optimization problem, using LLMs as optimizers and evaluators, can effectively generate high-quality, self-aligned instructions.

Summary by alphaXiv.