Reproduce “The Effectiveness and Durability of Pretraining Data Filtering in Protein Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Aaron Maiwald, Roderick McNeill, Jaeyoung Lee, Matthew A. Clarke, Lennart Justen, Alejandro Acelas, Ivo Andrews, Giles Harper-Donnelly and 4 more
The authors test whether withholding viral sequences from protein-model pretraining selectively reduces measured viral-protein performance. They train models without viral sequences, evaluate prediction tasks, then fine-tune on withheld data. In the 3B model, filtering raises viral masked-residue prediction loss by 0.173 nats, while nonviral changes remain within ±0.008 nats. Fine-tuning restores performance; the study tests prediction, not pathogen design.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.