Reproduce “ScamWorld: An Interactive Environment for Evaluating Web Agents Against Online Fraud” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yutao Wu, Xiao Liu, Ming Wen, Hanxun Huang, Jun Wang, Xingjun Ma
ScamWorld evaluates web agents on reconstructed scam websites, making documented fraud signs visible before the final action. It includes 363 scam websites and 53 legitimate controls. Across six frontier-model agents, deception rates were 28% for payment tasks and 2% for tasks involving credentials, card details, or wallet access; these controlled benchmark rates do not establish real-world fraud rates.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.