Reproduce “ProteinJEPA: Latent prediction improves protein language model pretraining” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Dan Ofer, Dafna Shahaf, Michal Linial
ProteinJEPA adds latent-representation prediction to masked amino-acid training for protein language models. A student predicts masked residues and a teacher’s hidden representation of the same unmasked protein. In continued pretraining, ESM2-150M reached 0.507 Recall@1 on SCOPe-40 retrieval, versus 0.404 for compute-matched MLM-only continuation; gains varied across tasks and were weaker from random initialization.
Key insights
- Latent prediction can complement residue prediction: removing MLM and training with JEPA alone causes large downstream losses, while adding JEPA can improve representations without improving token-prediction loss.
- The clearest benefits appear on structure- and homology-sensitive tasks, consistent with—but not proving—the idea that intermediate-layer targets preserve useful biological relationships.
- Results depend on training context: gains are most reproducible when continuing pretraining from an existing ESM2 checkpoint; evidence from random initialization and other backbones is less consistent.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.