Reproduce “CoPeP: Benchmarking Continual Pretraining for Protein Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Darshan Patil, Pranshu Malviya, Mathieu Reymond, Quentin Fournier, Sarath Chandar
CoPeP benchmarks continual protein-language-model training across ten annual UniRef100 releases from 2015–2024, totaling 580 million deduplicated entries. It uses release intersections and Temporal Replay, which preferentially replays persistent sequences. The 2022∩2024 intersection reached validation pseudo-perplexity 7.29 versus 7.48 for unfiltered 2024 using 64% as much data, while update methods varied by evaluation. No compute savings were measured.
Key insights
- Persistence across database releases is informative: a model trained on sequences shared between the 2022 and 2024 releases improves validation perplexity by about 2.5% while using only 64% of the full 2024 dataset.
- Continual-learning methods have task-dependent strengths rather than a single universal winner. Replay favors modeling the persistent natural protein distribution, whereas unlearning and plasticity methods can better support mutation prediction and transfer.
- Temporal metadata can act as a proxy for data reliability, but it also reflects curation practices and research attention, so persistence should not be interpreted as a direct measure of biological validity.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.