Reproduce “ScamWorld: An Interactive Environment for Evaluating Web Agents Against Online Fraud” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yutao Wu, Xiao Liu, Ming Wen, Hanxun Huang, Jun Wang, Xingjun Ma

ScamWorld evaluates web agents on reconstructed scam websites, making documented fraud signs visible before the final action. It includes 363 scam websites and 53 legitimate controls. Across six frontier-model agents, deception rates were 28% for payment tasks and 2% for tasks involving credentials, card details, or wallet access; these controlled benchmark rates do not establish real-world fraud rates.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.