Reproduce “UserArena: Benchmarking Interactive User Simulation for Long-Horizon Shopping Agents” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jujia Zhao, Jiakai Tang, Jing Yu, Wen Chen, Jian Wu, Han Zhu, Fei Sun, Suzan Verberne and 1 more
UserArena pairs an interactive shopping benchmark with On-Policy Trajectory Distillation (OPTD) to evaluate simulators across visits. OPTD summarizes a training trajectory and uses trust-region-constrained teacher guidance on the student’s own responses, grounded in recorded behavior. Across sessions, OPTD improves selected fidelity measures over STA; for one ranking-policy pair, its simulated effect is 0.713× online, underestimating it by 28.7%.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.