Reproduce “Ensemble Learning for Heterogeneous Large Language Models with Deep Parallel Collaboration” on OpenResearch

About this paper

Yichong Huang, Xiaocheng Feng, Baohang Li, Yang Xiang, Hui Wang, Bing Qin, Ting Liu

Researchers from Harbin Institute of Technology and Peng Cheng Laboratory developed DEEPEN, a training-free ensemble framework that fuses the probability distributions of heterogeneous Large Language Models by transforming their outputs into a universal relative semantic space, overcoming vocabulary misalignment. This approach consistently improved performance across various benchmarks compared to individual models and other ensemble baselines, demonstrating a robust method for leveraging diverse LLM strengths.

Key insights

• The concept of "relative representation," using cosine similarity to anchor tokens, provides a model-invariant space for comparing token semantics across heterogeneous vocabularies.
• Effective fusion of internal probability distributions from diverse LLMs (dense, sparse, generalist, specialist) is achievable in a training-free manner by transforming to and from a universal relative space.
• Normalizing relative representation matrices and carefully tuning the iterative search for inverse transformation are critical for the framework's robust performance.

Summary by alphaXiv.