Reproduce “Bootstrapping your behavior: a new pretraining strategy for user behavior sequence data” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Weichang Wu, Xiaolu Zhang, Jun Zhou, Yuchen Li, Wenwen Xia
Bootstrapping Your Behavior (BYB) introduces a pretraining strategy for User Behavior Sequences that automatically constructs a supervision embedding from future behaviors, moving beyond manual vocabulary selection. This approach yields an average AUC improvement of 3.9% over prior methods in offline tasks and reduces bad debt risk by 2.7-7.1% in Kolmogorov-Smirnov score during online deployment at Ant Group, while nearly doubling training throughput.
Key insights
- Automated construction of supervision embeddings for future behaviors, rather than relying on manual vocabularies, allows for more comprehensive and less biased learning of user representations.
- An Exponential Moving Average (EMA) strategy for updating the teacher encoder is essential for providing stable and high-quality supervisory signals during self-supervised pretraining.
- Behavior pooling effectively addresses the unique challenges of User Behavior Sequence data, such as local randomness and long sequences, drastically improving computational efficiency.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.