Reproduce “Scaling Laws for Behavioral Foundation Models over User Event Sequences” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Rickard Brüel Gabrielsson
The study quantifies optimal resource allocation for key hyperparameters in a two-part behavioral foundation model architecture, revealing distinct scaling laws for event embedder capacity, batch size, model/data balance, and negative sampling compared to language models. It identifies a compute-optimal embedder share of about 2% and shows how the optimal data-to-parameter ratio evolves towards Chinchilla-like values at higher compute budgets.
Key insights
- The compute-optimal event embedder capacity is consistently small (approximately 2% of total model parameters), driven by compute and effective-epoch asymmetries.
- The 'optimal' training recipe (e.g., critical batch size, negative sample count) is highly dependent on the chosen evaluation metric, and the correlation between training loss and ranking quality can flip at higher compute.
- Behavioral scaling is initially data-heavy, with a higher optimal data-to-parameter ratio than LMs, but this ratio decreases monotonically with increasing compute, approaching the Chinchilla heuristic at larger scales.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.