Reproduce “SimAB: Simulating A/B Tests with Persona-Conditioned AI Agents for Rapid Design Evaluation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tim Rieder, Marian Schneider, Mario Truss, Vitaly Tsaplin, Alina Rublea, Sinem Dere, Francisco Chicharro Sanz, Tobias Reiss and 1 more
Researchers from ETH Zurich and Adobe Research developed SimAB, a system that simulates A/B tests using persona-conditioned AI agents to generate synthetic preference judgments for design variants. This approach allows for rapid design evaluation, achieving a 67% overall agreement with historical A/B test outcomes and reducing feedback time from months to minutes, particularly for scenarios with limited live traffic.
Key insights
- Persona diversity is crucial for accurate simulation, with moderate diversity proving effective, while a single persona significantly degrades performance.
- Bias mitigation techniques like counterbalancing and neutral naming are critical for preventing LLM positional biases from influencing simulation results.
- Integrating contextual documents, such as analytics reports, can substantially accelerate the time to reach statistical significance in simulations while maintaining accuracy.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.