Reproduce “Birbal: An efficient 7B instruct-model fine-tuned with curated datasets” on OpenResearch

About this paper

Ashvini Kumar Jindal, Pawan Kumar Rajpoot, Ankur Parikh

The Birbal model, a 7B instruct-model, achieved first place in the NeurIPS LLM Efficiency Challenge's RTX 4090 track by fine-tuning Mistral-7B on a single consumer-grade GPU with curated open-source datasets. It outperformed the second-best submission by over 35% in the final weighted score, showcasing the effectiveness of data-centric approaches under strict resource constraints.

Key insights

• Strategic data curation, emphasizing quality and diversity, can be more effective than sheer data quantity or extensive hardware for achieving high-performance LLM fine-tuning.
• Top-tier LLM performance is achievable on accessible, consumer-grade hardware (single RTX 4090 GPU), significantly democratizing access to advanced LLM capabilities.
• There are performance trade-offs with increasing dataset sizes during instruction fine-tuning, as larger datasets can improve performance on some tasks while reducing it on others.

Summary by alphaXiv.