Findings from the audits and the checks: what AI agents are actually served, where sites silently fail, and what to do about it.
OpenAI runs three crawlers with three different jobs: training, search indexing and live fetching. Blocking one does not block the others, and most robots.txt policies only address one of the three.
Since July 2025 new Cloudflare domains block AI training crawlers by default, and three further settings can cut AI agents off entirely. We hit all three setting up our own site. Here is how to check yours, in five steps.