Cloudflare’s security team announced it has found evidence that Perplexity’s web crawler is evading site owners’ requests to block AI bots. The infrastructure provider began investigating after customers reported unexpected scraping of their pages. In its August 4 technical report, Cloudflare said its researchers observed Perplexity changing its user-agent string and even its network identifiers to mimic legitimate browsers. These tactics allowed the crawler to slip past robots.txt rules and custom blocks meant to stop data harvesting.
How Perplexity Crawls Blocked Sites
Cloudflare explains that Perplexity’s crawler first identifies itself with a declared user-agent. When blocked, it switches to a generic “Mozilla Chrome on macOS” signature. It also moves its traffic across multiple autonomous system numbers to avoid IP-based defenses. Cloudflare used machine learning and network analysis to fingerprint the crawler’s behavior across millions of daily requests to tens of thousands of domains. In response, Cloudflare has removed Perplexity from its verified crawler list and updated its own tools to block these stealth requests.
Perplexity’s Response and Dispute
Perplexity spokesperson Jesse Dwyer dismissed the report as a “sales pitch” and insisted the screenshots showed no actual content was retrieved. In a follow-up, Dwyer claimed the bot Cloudflare identified “is not even ours.” Cloudflare stands by its findings, noting that tests confirmed Perplexity circumvented explicit no-crawl settings. Perplexity has previously faced accusations. Last year, Wired reported the company was plagiarizing news articles, and its CEO struggled to define plagiarism when pressed at a conference.
The Broader Battle Over AI Scraping
As AI services like Perplexity, ChatGPT, and Google Bard rely on vast web data, publishers and site operators have fought back. The robots.txt standard can signal “do not crawl” to search engines and bots. Yet AI companies often ignore these conventions, arguing that public web data is fair use for training models. In May, Cloudflare launched a marketplace letting publishers charge scrapers for access. It also released free tools to help sites block unwanted AI crawlers. CEO Matthew Prince has warned that unregulated scraping threatens the business model of online publishers.
What Website Owners Can Do
Cloudflare sites have new options to reject stealth crawlers automatically. Those publishers who have not implemented Cloudflare may use sophisticated bot-management tools or clear-cut server policies in order to implement no-crawl policies. They are able to filter the traffic to look out for signs of the deceitful user-agent spoof, and they are in a position to inhibit requests that resemble the familiar modes of disguise. Incidents in some areas may also take legal actions through terms of service breaches or information protection legislation.
The results provided by Cloudflare point to the conflict between AI innovation and the respect for the preferences of the publishers. With the tactics of data collection maturing on the side of the AI tools, site owners have to remain alert to ensure threats to their content and revenue collection.
