On Wednesday, the company filed a complaint in California stating that Anthropic developer of Claude ignores Robots Exclusion Protocol REP, or robots.txt, which blocks AI crawlers from scraping a site's content. Research indicates that other AI companies are also engaging in this practice: In March, Columbia's Tow Center found that multiple chatbots, including Perplexity, could still retrieve articles from publishers that had blocked their crawlers using...
