Reproduce “RouterDC: Query-Based Router by Dual Contrastive Learning for Assembling Large Language Models” on OpenResearch

About this paper

Shuhao Chen, Weisen Jiang, Baijiong Lin, James T. Kwok, Yu Zhang

Researchers from SUSTech and HKUST developed RouterDC, a query-based router for Large Language Models that utilizes a novel dual contrastive learning framework. The system efficiently selects the most appropriate LLM for a given input query, achieving 58.54% average accuracy on in-distribution tasks, a +3.98% improvement over the best individual LLM, and operates approximately 6 times faster than traditional ensembling methods.

Key insights

• Dual contrastive learning, specifically the Sample-LLM Contrastive Loss, effectively addresses the 'multiple expert' problem by allowing the router to pull query embeddings closer to multiple top-performing LLMs, providing a stronger supervisory signal than traditional distribution alignment.
• The Sample-Sample Contrastive Loss significantly improves routing consistency and model stability by ensuring that similar queries are mapped to similar embeddings, even when relying on unsupervised clustering for query grouping.
• The training method, which avoids backpropagation through large frozen LLMs and requires few training samples per task, contributes to RouterDC's efficiency and data effectiveness.

Summary by alphaXiv.