Reproduce “Topology-Enhanced Alignment for Large Language Models: Trajectory Topology Loss and Topological Preference Optimization” on OpenResearch

About this paper

Yurui Pan, Ke Xu, Bo Peng

Researchers from Fudan University, Tongji University, and Shanghai Ocean University developed a framework that enhances large language model (LLM) alignment by incorporating topological information from hidden state trajectories. This approach, which includes Trajectory Topology Loss (TTL) for supervised fine-tuning and Topological Preference Optimization (TPO) for preference alignment, consistently improved reward-model scores and helpfulness ratings by 2-4 percentage points across different LLM architectures like Qwen2.5-7B-Instruct and Llama-3-8B-Instruct.

Key insights

• Explicitly supervising the semantic trajectories in an LLM's hidden space using topological information can lead to more robust and effective alignment with human preferences.
• 0-dimensional persistent homology provides valuable global connectivity information that helps derive meaningful 'bridges' between prompts and gold answers, or between rejected and chosen responses.
• The combination of topic-aware preference vectors and dynamic weighting schemes is crucial for stabilizing training and maximizing performance gains in topology-enhanced preference optimization.

Summary by alphaXiv.