Several website operators have accused the company Anthropic and its chatbot Claude of excessively crawling their websites, disrupting normal operations. Matt Barrie, CEO of Freelancer.com, claims that Anthropic is “by far the most aggressive crawler” of his freelance platform.
Please follow us on Twitter and Facebook
According to data provided to the Financial Times, Freelancer.com received 3.5 million visits from a web crawler connected to Anthropic in just four hours. This amount of data reportedly far exceeds that of other AI crawlers. Barrie estimates that Anthropic’s crawling activity is “probably about five times that of the second largest AI crawler.”
The problem is not limited to Freelancer.com. Kyle Wiens, CEO of iFixit.com, reported that his repair website received one million hits from Anthropic bots within 24 hours.
Impacts and Responses to AI Website Scraping
Anthropic sees itself as a developer of responsible AI systems. The company said it is investigating the reported incidents and respecting the wishes of publishers, reports the Financial Times. The AI startup claims to adhere to standard web protocols such as robots.txt, which govern crawler behavior, but some website operators report that their attempts to deny access to Anthropic’s bots have been unsuccessful.
Eric Holscher, co-founder of document hosting website Read the Docs, pointed out the financial impact of AI crawlers, citing the significant cost of traffic in a recent blog post. As AI companies compete to develop more powerful models, they are looking for new sources of training data. Some, like OpenAI, have partnered with content providers. Anthropic has not yet announced any similar agreements.
Read Also: According to the AI Act, Meta’s New Llama 3.1s Are a “Systemic Risk”: The Issue with TFLOPS