Sites that block AI training crawlers but allow AI search crawlers
21 of the 303 sites we read on .
These sites disallow at least one training crawler (GPTBot, ClaudeBot, Google-Extended, CCBot or Applebot-Extended) while OAI-SearchBot, Claude-SearchBot and PerplexityBot stay allowed on the homepage: opted out of training, still eligible to be found and cited by ChatGPT, Claude and Perplexity.
| Site | Blocked training crawlers |
|---|---|
| airbnb.comConsumer and e-commerce | GPTBot, ClaudeBot, Applebot-Extended |
| calendly.comProductivity | CCBot |
| canva.comDeveloper tools and SaaS | GPTBot, ClaudeBot, CCBot, Applebot-Extended |
| crunchbase.comStartup and review sites | GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended |
| figma.comDeveloper tools and SaaS | GPTBot, ClaudeBot, Google-Extended, CCBot |
| glassdoor.comConsumer and e-commerce | GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended |
| goodreads.comEntertainment | GPTBot, CCBot |
| healthline.comHealth reference | GPTBot, ClaudeBot, CCBot, Applebot-Extended |
| letterboxd.comEntertainment | GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended |
| loom.comProductivity | GPTBot |
| midjourney.comAI companies and tools | GPTBot, Google-Extended, CCBot, Applebot-Extended |
| netflix.comConsumer and e-commerce | CCBot, Applebot-Extended |
| snapchat.comSocial and community | GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended |
| techcrunch.comNews and media | GPTBot, ClaudeBot, CCBot, Applebot-Extended |
| together.aiAI companies and tools | Google-Extended |
| tripadvisor.comConsumer and e-commerce | GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended |
| trustpilot.comStartup and review sites | Applebot-Extended |
| vimeo.comSocial and community | CCBot |
| webmd.comHealth reference | GPTBot, ClaudeBot, CCBot |
| who.intEducation and reference | CCBot |
| zillow.comConsumer and e-commerce | Applebot-Extended |
Not on the list, or not sure?
Check your own site with the free llms.txt checker or the AI Crawler Access Checker. No signup. All data: 303 sites, CSV.
The sample is 303 popular, mostly English-language sites chosen by SEOBolt, not a random sample of the web. robots.txt and llms.txt show what a site publishes, not what any crawler does. Removal or correction: support@seobolt.app. SEOBolt builds the audit tool this data comes from.