Reproduce “Beyond ESM2: Graph-Enhanced Protein Sequence Modeling with Efficient Clustering” on OpenResearch

About this paper

Shujian Jiao, Bingxuan Li, Lei Wang, Xiaojin Zhang, Wei Chen, Jiajie Peng, Zhongyu Wei

ComproESM, a model developed by researchers from Huazhong University of Science and Technology, Fudan University, and Northwestern Polytechnical University, refines protein sequence representations by integrating hierarchical protein family information via a graph-based clustering algorithm with masked language modeling. The model consistently outperformed ESM2-12layer on six downstream tasks, including a 3.57 percentage point increase in protein family classification accuracy and a 2.8 percentage point improvement in homology detection.

Key insights

• Explicitly incorporating hierarchical protein family data into pre-training enhances protein representations, leading to better functional interpretability and improved task performance.
• A synergistic learning objective combining local amino acid context (MLM) with global hierarchical functional context (graph-based community propagation) produces more robust protein representations.
• The Community Propagation-Based Clustering Algorithm provides a resource-efficient method for integrating multi-category hierarchical labels into protein language models via graph pre-training.

Summary by alphaXiv.