Reproduce “Scaling Laws for the Value of Individual Data Points in Machine Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ian Covert, Wenlong Ji, Tatsunori Hashimoto, James Zou
Researchers at Stanford University introduce and validate individualized data scaling laws, which characterize how the marginal contribution of a single data point to a model's performance scales with overall dataset size. They developed efficient estimation methods and demonstrated how these laws can be used for dynamic data valuation and context-dependent data subset selection.
Key insights
- The marginal contributions of individual data points to model performance follow a predictable log-linear scaling behavior with respect to the logarithm of dataset size.
- The data-point-specific scaling exponent $\alpha(z)$ is highly heterogeneous, indicating that the rate at which a data point's value decays varies significantly across examples, leading to dynamic changes in their relative importance.
- A data point's proximity to the decision boundary correlates with its scaling exponent, suggesting that points closer to the boundary retain their relative utility more effectively in larger datasets.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.