Image missing.
Cloudflare will now, by default, block AI bots from crawling its clients’ websites

Peter Hall

created: July 1, 2025, 10 a.m. | updated: July 7, 2025, 12:27 p.m.

Clients can also set a rate for how much it will cost AI bots to crawl their website. Cloudflare already has a bot verification system where AI web crawlers can tell websites who they work for and what they want to do. For these, Cloudflare hopes its system can facilitate good-faith negotiations between AI companies and website owners. For the less honest crawlers, Cloudflare plans to use its experience dealing with coordinated denial-of-service attacks from bots to stop them. While this approach will still apply for the truly bad actors, the company says it hopes its new services can foster better relationships between AI companies and content producers.

5 months, 2 weeks ago: MIT Technology Review