Sites whose robots.txt blocks ChatGPT, Claude or Perplexity search crawlers
23 of the 303 sites we read on .
These sites disallow at least one of OAI-SearchBot, Claude-SearchBot or PerplexityBot on the homepage. The matching assistant has less of the site to index and cite. Many publishers do this deliberately.
| Site | Blocked search crawlers |
|---|---|
| amazon.comConsumer and e-commerce | OAI-SearchBot, Claude-SearchBot, PerplexityBot |
| apnews.comNews and media | Claude-SearchBot, PerplexityBot |
| bloomberg.comNews and media | PerplexityBot |
| cnn.comNews and media | OAI-SearchBot, Claude-SearchBot, PerplexityBot |
| ebay.comConsumer and e-commerce | PerplexityBot |
| facebook.comSocial and community | OAI-SearchBot, Claude-SearchBot, PerplexityBot |
| forbes.comNews and media | PerplexityBot |
| imdb.comEntertainment | OAI-SearchBot, Claude-SearchBot, PerplexityBot |
| instacart.comTravel, jobs and marketplaces | Claude-SearchBot, PerplexityBot |
| instagram.comSocial and community | OAI-SearchBot, Claude-SearchBot, PerplexityBot |
| npr.orgNews and media | OAI-SearchBot, Claude-SearchBot, PerplexityBot |
| nytimes.comNews and media | OAI-SearchBot, Claude-SearchBot, PerplexityBot |
| pinterest.comSocial and community | OAI-SearchBot, Claude-SearchBot, PerplexityBot |
| quizlet.comEducation and reference | PerplexityBot |
| quora.comEducation and reference | OAI-SearchBot, Claude-SearchBot, PerplexityBot |
| reuters.comNews and media | Claude-SearchBot, PerplexityBot |
| theguardian.comNews and media | Claude-SearchBot, PerplexityBot |
| theverge.comNews and media | Claude-SearchBot, PerplexityBot |
| tiktok.comSocial and community | OAI-SearchBot, Claude-SearchBot, PerplexityBot |
| twitter.comSocial and community | OAI-SearchBot, Claude-SearchBot, PerplexityBot |
| washingtonpost.comNews and media | PerplexityBot |
| whatsapp.comSocial and community | PerplexityBot |
| x.comSocial and community | OAI-SearchBot, Claude-SearchBot, PerplexityBot |
Not on the list, or not sure?
Check your own site with the free llms.txt checker or the AI Crawler Access Checker. No signup. All data: 303 sites, CSV.
The sample is 303 popular, mostly English-language sites chosen by SEOBolt, not a random sample of the web. robots.txt and llms.txt show what a site publishes, not what any crawler does. Removal or correction: support@seobolt.app. SEOBolt builds the audit tool this data comes from.