
Cloudflare Is Blocking AI Crawlers by Default
Kate Knibbs
created: July 1, 2025, 10 a.m. | updated: July 3, 2025, 12:46 p.m.
Last year, internet infrastructure firm Cloudflare launched tools enabling its customers to block AI scrapers.
It has switched to blocking AI crawlers by default for its customers and is moving forward with a Pay Per Crawl program that lets customers charge AI companies to scrape their websites.
Even when websites can handle the heightened activity, many do not want AI crawlers scraping their content, especially news publications that are demanding AI companies to pay to use their work.
Cloudflare also says it can identify even “shadow” scrapers that are not publicized by AI companies.
Meanwhile, there’s an entire online ecosystem of tutorials about how to evade Cloudflare’s bot blocking tools aimed at web scrapers.
1 month ago: WIRED