Sites whose robots.txt blocks ChatGPT, Claude or Perplexity search crawlers

23 of the 303 sites we read on .

These sites disallow at least one of OAI-SearchBot, Claude-SearchBot or PerplexityBot on the homepage. The matching assistant has less of the site to index and cite. Many publishers do this deliberately.

Sites that block AI search crawlers
SiteBlocked search crawlers
amazon.comConsumer and e-commerceOAI-SearchBot, Claude-SearchBot, PerplexityBot
apnews.comNews and mediaClaude-SearchBot, PerplexityBot
bloomberg.comNews and mediaPerplexityBot
cnn.comNews and mediaOAI-SearchBot, Claude-SearchBot, PerplexityBot
ebay.comConsumer and e-commercePerplexityBot
facebook.comSocial and communityOAI-SearchBot, Claude-SearchBot, PerplexityBot
forbes.comNews and mediaPerplexityBot
imdb.comEntertainmentOAI-SearchBot, Claude-SearchBot, PerplexityBot
instacart.comTravel, jobs and marketplacesClaude-SearchBot, PerplexityBot
instagram.comSocial and communityOAI-SearchBot, Claude-SearchBot, PerplexityBot
npr.orgNews and mediaOAI-SearchBot, Claude-SearchBot, PerplexityBot
nytimes.comNews and mediaOAI-SearchBot, Claude-SearchBot, PerplexityBot
pinterest.comSocial and communityOAI-SearchBot, Claude-SearchBot, PerplexityBot
quizlet.comEducation and referencePerplexityBot
quora.comEducation and referenceOAI-SearchBot, Claude-SearchBot, PerplexityBot
reuters.comNews and mediaClaude-SearchBot, PerplexityBot
theguardian.comNews and mediaClaude-SearchBot, PerplexityBot
theverge.comNews and mediaClaude-SearchBot, PerplexityBot
tiktok.comSocial and communityOAI-SearchBot, Claude-SearchBot, PerplexityBot
twitter.comSocial and communityOAI-SearchBot, Claude-SearchBot, PerplexityBot
washingtonpost.comNews and mediaPerplexityBot
whatsapp.comSocial and communityPerplexityBot
x.comSocial and communityOAI-SearchBot, Claude-SearchBot, PerplexityBot

Not on the list, or not sure?

Check your own site with the free llms.txt checker or the AI Crawler Access Checker. No signup. All data: 303 sites, CSV.

The sample is 303 popular, mostly English-language sites chosen by SEOBolt, not a random sample of the web. robots.txt and llms.txt show what a site publishes, not what any crawler does. Removal or correction: support@seobolt.app. SEOBolt builds the audit tool this data comes from.