Reproduce “Protein Language Model Zero-Shot Fitness Predictions are Improved by Inference-only Dropout” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Aditya Ravuri, Neil D. Lawrence
Researchers at the University of Cambridge present a method that improves zero-shot protein fitness predictions from Protein Language Models by injecting dropout layers only during inference. This approach consistently enhanced Spearman rank correlation coefficients across ESM2 model sizes, with a notable benefit for the smaller 35M parameter model, without requiring model retraining.
Key insights
- Dropout can improve Protein Language Model (PLM) performance even when applied solely during inference, without requiring the model to be trained with dropout.
- Strategic placement of dropout, such as between the embedding layer and transformer blocks, and within early transformer layers for larger models, is key to maximizing performance gains.
- The observed performance increase is hypothesized to be due to improved model calibration and the method's ability to better handle out-of-domain protein sequences.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.