Reproduce “Identifying AI Web Scrapers Using Canary Tokens” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Steven Seiden, Triss Ren, Caroline Zhang, Taein Kim, Enze Liu, Emily Wenger
The authors present a method to trace scraper-associated content reaching chatbots. They give visitors different facts, record each visitor’s User-Agent and ASN, then query chatbots and match returned facts to those records. Of 18 matched systems, 10 returned content associated with third-party search agents. After crawling, 12 still returned associated content when sites were offline or robots.txt-restricted; this does not test initial blocking.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.