Reproduce “Proving and Improving the Platonic Representation Hypothesis with JEPAs” on OpenResearch

About this paper

Randall Balestriero, Yann LeCun

The authors test whether positive views can align separately trained text and image representations. A JEPA trained on different descriptions of the same scene learns stronger alignment than one trained on corruptions of a single description. On WIT-1024, the 109M-parameter model scores .184 against DINOv2 ViT-B, exceeding Qwen2.5-72B’s .154; the theorem’s idealized assumptions limit generalization.

Key insights

• Alignment depends not only on model scale but on which underlying states the training objective treats as related; shared positive-pair structure can make independently learned embeddings compatible.
• Positive pairs that connect separate observations of the same scene encourage representations of shared content, while pairs that preserve nuisance details can consume embedding dimensions and limit cross-modal alignment.
• Next-token prediction can be interpreted as having a JEPA-like positive relation between prefixes and continuations, but this does not explicitly group text by shared scenes; the analogy does not make the theoretical results directly applicable to language models.

Summary by alphaXiv.