Reproduce “Sloth: scaling laws for LLM skills to predict multi-benchmark performance across families” on OpenResearch

About this paper

Felipe Maia Polo, Seamus Somerstep, Leshem Choshen, Yuekai Sun, Mikhail Yurochkin

The "Sloth" paper introduces Skills Scaling Laws (SSLaws) to accurately predict large language model performance across multiple benchmarks and model families. It models LLM capabilities as low-dimensional latent skills influenced by computational resources, achieving an average mean-absolute-error (MAE) of 3.3-4.1% on Open LLM Leaderboard benchmarks, outperforming baselines.

Key insights

• LLM performance can be effectively decomposed into a small set of interpretable latent skills, which provides a more efficient and accurate way to model and predict performance.
• While the influence of computational resources on latent skills is broadly similar across families, the efficiency with which each family converts these resources into higher skill levels can vary.
• Specific latent skills (e.g., Reasoning, Knowledge, Instruction Following) are differentially impacted by model size, training tokens, and instruction tuning, offering insights into optimal resource allocation for desired capabilities.

Summary by alphaXiv.