Reproduce “BehaviorBench: Modeling Real-World User Decisions from Behavioral Traces” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Liangwei Yang, Jielin Qiu, Zixiang Chen, Ming Zhu, Juntao Tan, Zhiwei Liu, Wenting Zhao, Zhujun Lan and 4 more
BEHAVIORBENCH introduces a benchmark for evaluating personalized decision modeling directly from real-world behavioral traces, addressing limitations of simulated user data. It defines two prediction tasks—Belief and Trade—and systematically evaluates generative models on their ability to utilize diverse forms of user history, consistently showing personalization gains across tasks.
Key insights
- Behavioral history consistently provides personalization gains across both Belief and Trade prediction tasks, demonstrating the usability of prior user behavior for personalized modeling.
- The effectiveness of history representation is task-specific; aggregated user profiles (ProfileGen) are more effective for stable Belief prediction, while direct sequential history (DirectGen) is better for dynamic Trade prediction.
- Model ranking depends on the specific metric (accuracy vs. continuous error), indicating that performance encompasses both directional correctness and calibration, with frontier models excelling in accuracy but open-weight models being competitive in calibration.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.