[vendor]
Cloudflare Blog — AI[vendor]By Bryan Becker
Have it both ways: stay discoverable in search while disallowing AI training
Why it matters
Site owners can now refuse AI training by major mixed-use crawlers without disappearing from Google, Apple, or Microsoft search results.
The brief
5 points
- Cloudflare's Disallow AI Training setting lets sites stay search-indexed while blocking the same crawler from training on their content.
- Apple, Google, and Microsoft honor the setting or have committed to honor it within a specified time frame.
- Seventeen percent of Cloudflare sites enable some mechanism to block AI training; fewer than 1% block search bots.
- By early next year, Cloudflare aims to let sites control how much of their content appears in AI summaries.
- Cloudflare argues robots.txt alone cannot identify who is crawling, determine why, or stop a crawler that ignores it.