Reproduce “Output Dilution: Redundant but Fragile Representations in MoE Models” on OpenResearch

About this paper

Orion Reblitz-Richardson

This research from Distiller Labs examines how Mixture-of-Experts (MoE) models encode moral content, finding that despite achieving high probing accuracy, their representations are 4.2 times more fragile than dense models due to an "output dilution" effect where MoE block outputs are significantly smaller in scale. The study concludes that moral information is uniformly distributed across MoE experts and layers, without specialization.

Key insights

• MoE models exhibit high probing accuracy for moral content, yet show significantly higher representational fragility (4.2x greater) compared to dense models.
• Moral content is encoded uniformly across all experts and layers in MoE models; no specialized "moral experts" or content-specific router preferences were identified.
• The observed fragility is due to "output dilution," where the MoE block's aggregated output is, on average, 74 times smaller in scale than dense MLP outputs, making the signal highly susceptible to noise despite being decodable by probes.

Summary by alphaXiv.