Reproduce “Shopping Reasoning Bench: An Expert-Authored Benchmark for Multi-Turn Conversational Shopping Assistants” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shuxian Fan, Seonwoo Min, Youna Hu, Botao Xia, Jayakrishnan Unnikrishnan, Rowan Musselmann, Yifan Gao, Qingyu Yin and 2 more
The 'Shopping Reasoning Bench,' a new expert-authored benchmark from Amazon, provides a comprehensive, multi-turn evaluation for AI shopping assistants, utilizing a novel taxonomy and granular rubrics. It reveals that current frontier LLMs achieve overall pass rates between 57.4% and 77.2%, consistently struggling to provide expert-level, actionable advice, especially in multi-turn interactions.
Key insights
- The benchmark is unsaturated, with top frontier LLMs achieving only 57-77% pass rates, indicating significant room for improvement in AI shopping assistants.
- Models consistently struggle with 'optional' rubrics (expert-level advice) compared to 'required' rubrics (baseline correctness), showing a 13-29 percentage point performance gap.
- Multi-turn conversations lead to performance degradation in all frontier models, underscoring challenges in maintaining coherence and context over extended dialogues.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.