Reproduce “Energy Efficient Protein Language Models: Leveraging Small Language Models with LoRA for Controllable Protein Generation” on OpenResearch

About this paper

Aayush Shah, Shankar Jayaratnam

Researchers from Esperanto Technologies developed energy-efficient protein language models by fine-tuning small LLMs (Llama 3 and Phi 3) with LoRA, demonstrating a 70% reduction in training time and robust performance in generating proteins with specified properties. The models achieved a 60% to over threefold increase in energy efficiency (TPS/W) for inference when deployed on the ET-SoC-1 chip compared to an NVIDIA A100 GPU.

Key insights

• Parameter-efficient fine-tuning with LoRA on smaller LLMs can yield robust protein generation capabilities, significantly reducing training resources and costs.
• Integrating specific enzyme classes into instruction-based fine-tuning effectively expands controllable generation to a broader range of functional protein properties.
• Specialized, low-power AI hardware offers substantial energy efficiency improvements for protein language model inference, paving the way for more sustainable AI deployment.

Summary by alphaXiv.