Reproduce “Universally Converging Representations of Matter Across Scientific Foundation Models” on OpenResearch

About this paper

Sathya Edamadaka, Soojung Yang, Ju Li, Rafael Gómez-Bombarelli

The authors compare hidden representations from 59 scientific models across molecular, materials, and protein datasets to test whether models using different inputs organize structures similarly. They measure local and alignment, dimensionality, and information imbalance. Molecular-string and 3D atomistic models align on QM9, while the pattern depends on distribution: weak models diverge on OMat24 but converge on less information for out-of-distribution OMol25.

Key insights

• Representational alignment provides a quantitative benchmark for diagnosing "foundational generality," offering a deeper assessment than task-specific performance metrics.
• The training data distribution often exerts a stronger influence on a model's representation space than its architectural inductive biases, particularly for in-distribution data.
• Regularization schemes can achieve similar representational benefits as explicit architectural equivariance at a significantly lower computational cost, challenging prior assumptions.

Summary by alphaXiv.