Sites that block AI training crawlers but allow AI search crawlers

21 of the 303 sites we read on .

These sites disallow at least one training crawler (GPTBot, ClaudeBot, Google-Extended, CCBot or Applebot-Extended) while OAI-SearchBot, Claude-SearchBot and PerplexityBot stay allowed on the homepage: opted out of training, still eligible to be found and cited by ChatGPT, Claude and Perplexity.

Sites that block AI training but allow AI search
SiteBlocked training crawlers
airbnb.comConsumer and e-commerceGPTBot, ClaudeBot, Applebot-Extended
calendly.comProductivityCCBot
canva.comDeveloper tools and SaaSGPTBot, ClaudeBot, CCBot, Applebot-Extended
crunchbase.comStartup and review sitesGPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended
figma.comDeveloper tools and SaaSGPTBot, ClaudeBot, Google-Extended, CCBot
glassdoor.comConsumer and e-commerceGPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended
goodreads.comEntertainmentGPTBot, CCBot
healthline.comHealth referenceGPTBot, ClaudeBot, CCBot, Applebot-Extended
letterboxd.comEntertainmentGPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended
loom.comProductivityGPTBot
midjourney.comAI companies and toolsGPTBot, Google-Extended, CCBot, Applebot-Extended
netflix.comConsumer and e-commerceCCBot, Applebot-Extended
snapchat.comSocial and communityGPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended
techcrunch.comNews and mediaGPTBot, ClaudeBot, CCBot, Applebot-Extended
together.aiAI companies and toolsGoogle-Extended
tripadvisor.comConsumer and e-commerceGPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended
trustpilot.comStartup and review sitesApplebot-Extended
vimeo.comSocial and communityCCBot
webmd.comHealth referenceGPTBot, ClaudeBot, CCBot
who.intEducation and referenceCCBot
zillow.comConsumer and e-commerceApplebot-Extended

Not on the list, or not sure?

Check your own site with the free llms.txt checker or the AI Crawler Access Checker. No signup. All data: 303 sites, CSV.

The sample is 303 popular, mostly English-language sites chosen by SEOBolt, not a random sample of the web. robots.txt and llms.txt show what a site publishes, not what any crawler does. Removal or correction: support@seobolt.app. SEOBolt builds the audit tool this data comes from.