Reproduce “Inferred Generative-Process Diversity Predicts Correlated Failure Across Language Models” on OpenResearch

About this paper

Ross Tieman, Evan Markou

The authors test whether output-sequence structure can indicate when two language models may fail differently. They compare compressed UTF-8 response strings with byte-shuffling controls across 38 models, then relate the residual to chance-corrected shared wrong-answer agreement on ten benchmark families. The mean partial Spearman association was −0.216. This is observational, pairwise evidence, not an ensemble intervention result.

Key insights

• Effective redundancy in multi-language model systems is contingent on "generative-process diversity" (differences in underlying generative mechanisms) rather than just surface-level semantic diversity in outputs.
• The permutation-residualized NCD (d_IGP) successfully infers this process-level diversity from observable outputs, capturing a distinct and more fundamental aspect of variation than semantic similarity.
• The order-specific component of compression diversity, isolated via permutation control, exhibits a negative association with correlated failure, contrasting with the mild positive association seen in raw NCD or its frequency-preserving component.

Summary by alphaXiv.