Reproduce “Representational Collapse in Multi-Agent LLM Committees: Measurement and Diversity-Aware Consensus” on OpenResearch

About this paper

Dipkumar Patel

This research quantifies "representational collapse" in multi-agent LLM committees, revealing that agents often produce highly similar latent reasoning representations despite distinct roles. It introduces DALC (Diversity-Aware Latent Consensus), a training-free protocol leveraging diversity weights from agent rationale embeddings to improve aggregation. On the GSM8K benchmark, DALC-Id achieved 87% accuracy, surpassing self-consistency by 3 percentage points while reducing token consumption by 26%.

Key insights

• Even with distinct role conditioning, homogeneous LLM agents exhibit significant "representational collapse," meaning their underlying reasoning representations are highly correlated rather than truly diverse.
• Representational collapse tends to be more severe on harder tasks (e.g., MATH-500 vs. GSM8K), suggesting models converge more tightly on challenging problems.
• The choice of embedding encoder is a critical design variable, as it modulates the detected severity of representational collapse and the downstream effectiveness of diversity-aware protocols.
• Aggressive orthogonalization (e.g., Gram-Schmidt) of embeddings can distort genuine reasoning differences, indicating that diversity weighting requires a nuanced approach to accurately reflect relevant distinctions.

Summary by alphaXiv.