Which top websites block ChatGPT, Claude and Perplexity?
We read the robots.txt, llms.txt and sitemap of 303 well-known websites on . Of the 271 whose robots.txt we could read, 34 block GPTBot (13%), 34 block ClaudeBot (13%) and 23 block PerplexityBot (8%). Only 12 block OpenAI's search crawler OAI-SearchBot (4%) against 34 for GPTBot: keeping out a training crawler and keeping out search are different choices.
Every site has its own page with the exact robots.txt rule behind each verdict. Download the CSV (free to reuse with a link back, CC BY 4.0).
Five kinds of robots.txt
226
Open to every AI crawler we test
No rule blocks any of the 12 crawlers on the homepage.
1
Blocks only crawlers that fetch a page on a user's request
ChatGPT-User, Claude-User or Perplexity-User is disallowed; the search and training crawlers are not.
21
Opted out of AI training, still open to AI search
Blocks training crawlers such as GPTBot, ClaudeBot or Google-Extended; ChatGPT, Claude and Perplexity search crawlers are allowed.
11
Blocks at least one AI search crawler
At least one of OAI-SearchBot, Claude-SearchBot or PerplexityBot is disallowed, so that assistant has less to cite.
12
Closed to AI search and training crawlers
All three search crawlers plus GPTBot and ClaudeBot are disallowed.
Of the 34 sites that block GPTBot, 28 name it in their own group and 6 only inherit a blanket User-agent: * disallow: for those, GPTBot is blocked because every crawler is, not because of an AI decision.
32 of 303 sites answered our robots.txt request with an error, a login wall or a non-robots page and are left out of every percentage on this page.
Crawler by crawler
| Crawler | What it does | Sites blocking it |
|---|---|---|
CCBotCommon Crawl | Model training | 37 of 271 (14%) |
GPTBotOpenAI | Model training | 34 of 271 (13%) |
ClaudeBotAnthropic | Model training | 34 of 271 (13%) |
Applebot-ExtendedApple | Model training | 34 of 271 (13%) |
Google-ExtendedGoogle | Model training | 24 of 271 (9%) |
PerplexityBotPerplexity | Search | 23 of 271 (8%) |
Claude-SearchBotAnthropic | Search | 17 of 271 (6%) |
Claude-UserAnthropic | User request | 17 of 271 (6%) |
Perplexity-UserPerplexity | User request | 17 of 271 (6%) |
ChatGPT-UserOpenAI | User request | 13 of 271 (5%) |
OAI-SearchBotOpenAI | Search | 12 of 271 (4%) |
ApplebotApple | Search | 3 of 271 (1%) |
Operator, purpose and user-agent tokens come from each vendor's own documentation; the sources are listed on the AI Crawler Access Checker.
By kind of site
| Category | Sites read | Block GPTBot | Block ClaudeBot | Block any AI search crawler | Publish llms.txt |
|---|---|---|---|---|---|
| Social and community | 13 | 8 (62%) | 8 (62%) | 7 (54%) | 0 of 13 (0%) |
| News and media | 14 | 8 (57%) | 11 (79%) | 10 (71%) | 2 of 11 (18%) |
| Consumer and e-commerce | 20 | 5 (25%) | 5 (25%) | 2 (10%) | 7 of 18 (39%) |
| Startup and review sites | 5 | 1 (20%) | 1 (20%) | 0 (0%) | 2 of 4 (50%) |
| Travel, jobs and marketplaces | 10 | 1 (10%) | 1 (10%) | 1 (10%) | 4 of 10 (40%) |
| Education and reference | 22 | 2 (9%) | 2 (9%) | 2 (9%) | 8 of 22 (36%) |
| Productivity | 13 | 1 (8%) | 0 (0%) | 0 (0%) | 7 of 13 (54%) |
| AI companies and tools | 30 | 1 (3%) | 0 (0%) | 0 (0%) | 15 of 30 (50%) |
| Developer tools and SaaS | 102 | 2 (2%) | 2 (2%) | 0 (0%) | 61 of 101 (60%) |
| Big tech brands | 5 | 0 (0%) | 0 (0%) | 0 (0%) | 3 of 5 (60%) |
| Finance | 14 | 0 (0%) | 0 (0%) | 0 (0%) | 4 of 12 (33%) |
| SEO and marketing tools | 16 | 0 (0%) | 0 (0%) | 0 (0%) | 10 of 16 (63%) |
Categories with fewer than five readable sites are not shown; llms.txt shares count only sites whose /llms.txt we could read. Our sample is popular, mostly English-language sites, not a random sample of the web.
All 303 sites
Showing 303 of 303 sites.
| Site | Blocked | GPTBot | ClaudeBot | PerplexityBot | llms.txt |
|---|---|---|---|---|---|
| 1password.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| activecampaign.comSEO and marketing tools | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| adidas.comConsumer and e-commerce | not readable | n/a | n/a | n/a | Not readable |
| ahrefs.comSEO and marketing tools | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| airbnb.comConsumer and e-commerce | 3 of 12 | Blocked | Blocked | Allowed | Not found |
| airtable.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| algolia.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Found |
| amazon.comConsumer and e-commerce | 10 of 12 | Blocked | Blocked | Blocked | Not found |
| amd.comBig tech brands | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| americanexpress.comFinance | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| angular.ioDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| anthropic.comAI companies and tools | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| apnews.comNews and media | 8 of 12 | Blocked | Blocked | Blocked | Not found |
| apple.comEducation and reference | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| arstechnica.comNews and media | not readable | n/a | n/a | n/a | Not readable |
| asana.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Found |
| astro.buildDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| atlassian.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Found |
| auth0.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Found |
| aws.amazon.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| azure.microsoft.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| backlinko.comSEO and marketing tools | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| bankofamerica.comFinance | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| basecamp.comProductivity | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| bbc.comNews and media | not readable | n/a | n/a | n/a | Not readable |
| beehiiv.comSEO and marketing tools | 0 of 12 | Allowed | Allowed | Allowed | Found |
| bestbuy.comConsumer and e-commerce | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| bigcommerce.comConsumer and e-commerce | 0 of 12 | Allowed | Allowed | Allowed | Found |
| binance.comFinance | 0 of 12 | Allowed | Allowed | Allowed | Not readable |
| bitwarden.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Found |
| bloomberg.comNews and media | 6 of 12 | Blocked | Blocked | Blocked | Not readable |
| bolt.newAI companies and tools | 0 of 12 | Allowed | Allowed | Allowed | Not found (HTML page) |
| booking.comConsumer and e-commerce | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| brevo.comSEO and marketing tools | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| brilliant.orgEducation and reference | 0 of 12 | Allowed | Allowed | Allowed | Found |
| buffer.comSEO and marketing tools | 0 of 12 | Allowed | Allowed | Allowed | Found |
| calendly.comProductivity | 1 of 12 | Allowed | Allowed | Allowed | Found |
| canva.comDeveloper tools and SaaS | 4 of 12 | Blocked | Blocked | Allowed | Not found |
| capterra.comStartup and review sites | not readable | n/a | n/a | n/a | Not readable |
| cdc.govHealth reference | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| character.aiAI companies and tools | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| chase.comFinance | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| chegg.comEducation and reference | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| clerk.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Found |
| clickup.comProductivity | 0 of 12 | Allowed | Allowed | Allowed | Found |
| cloud.google.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Not found (HTML page) |
| cloudflare.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Found |
| cloudinary.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Found |
| cnn.comNews and media | 10 of 12 | Blocked | Blocked | Blocked | Not found |
| cockroachlabs.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| codepen.ioDeveloper tools and SaaS | not readable | n/a | n/a | n/a | Not readable |
| codesandbox.ioDeveloper tools and SaaS | not readable | n/a | n/a | n/a | Not readable |
| cohere.comAI companies and tools | 0 of 12 | Allowed | Allowed | Allowed | Found |
| coinbase.comFinance | 0 of 12 | Allowed | Allowed | Allowed | Found |
| confluent.ioDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| contentful.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| convertkit.comSEO and marketing tools | not readable | n/a | n/a | n/a | Not readable |
| copy.aiAI companies and tools | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| coursera.orgEducation and reference | 0 of 12 | Allowed | Allowed | Allowed | Found |
| crates.ioDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| crunchbase.comStartup and review sites | 5 of 12 | Blocked | Blocked | Allowed | Not found |
| cursor.comAI companies and tools | 0 of 12 | Allowed | Allowed | Allowed | Found |
| databricks.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Found |
| datadoghq.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Found |
| deepl.comProductivity | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| deepseek.comAI companies and tools | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| dev.toNews and media | 0 of 12 | Allowed | Allowed | Allowed | Found |
| developer.mozilla.orgDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| digitalocean.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Found |
| discord.comSocial and community | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| disneyplus.comEntertainment | not readable | n/a | n/a | n/a | Not found (HTML page) |
| docker.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| doordash.comTravel, jobs and marketplaces | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| dropbox.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Found |
| duolingo.comEducation and reference | 0 of 12 | Allowed | Allowed | Allowed | Not found (HTML page) |
| ebay.comConsumer and e-commerce | 5 of 12 | Blocked | Blocked | Blocked | Not found |
| edx.orgEducation and reference | 0 of 12 | Allowed | Allowed | Allowed | Found |
| elastic.coDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Found |
| elevenlabs.ioAI companies and tools | 0 of 12 | Allowed | Allowed | Allowed | Found |
| etsy.comConsumer and e-commerce | 0 of 12 | Allowed | Allowed | Allowed | Found |
| evernote.comProductivity | 0 of 12 | Allowed | Allowed | Allowed | Found |
| expedia.comConsumer and e-commerce | 0 of 12 | Allowed | Allowed | Allowed | Found |
| facebook.comSocial and community | 11 of 12 | Blocked | Blocked | Blocked | Not found (HTML page) |
| fidelity.comFinance | 0 of 12 | Allowed | Allowed | Allowed | Not readable |
| figma.comDeveloper tools and SaaS | 4 of 12 | Blocked | Blocked | Allowed | Not found |
| fiverr.comTravel, jobs and marketplaces | 0 of 12 | Allowed | Allowed | Allowed | Found |
| fly.ioDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Found |
| forbes.comNews and media | 5 of 12 | Blocked | Blocked | Blocked | Not found |
| framer.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Found |
| freecodecamp.orgDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| freelancer.comTravel, jobs and marketplaces | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| g2.comStartup and review sites | 0 of 12 | Allowed | Allowed | Allowed | Found |
| getbootstrap.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Found |
| ghost.orgDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| github.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Found |
| gitlab.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Not readable |
| glassdoor.comConsumer and e-commerce | 5 of 12 | Blocked | Blocked | Allowed | Found |
| glitch.comDeveloper tools and SaaS | not readable | n/a | n/a | n/a | Not found (HTML page) |
| golang.orgDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| goodreads.comEntertainment | 2 of 12 | Blocked | Allowed | Allowed | Not found |
| google.comEducation and reference | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| grafana.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Found |
| grammarly.comProductivity | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| groq.comAI companies and tools | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| harvard.eduEducation and reference | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| hashnode.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Found |
| healthline.comHealth reference | 4 of 12 | Blocked | Blocked | Allowed | Not found |
| heroku.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Found |
| homedepot.comConsumer and e-commerce | not readable | n/a | n/a | n/a | Not readable |
| hootsuite.comSEO and marketing tools | 0 of 12 | Allowed | Allowed | Allowed | Found |
| hubspot.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Found |
| huggingface.coAI companies and tools | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| hulu.comEntertainment | not readable | n/a | n/a | n/a | Not found (HTML page) |
| ibm.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| ikea.comConsumer and e-commerce | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| imdb.comEntertainment | 11 of 12 | Blocked | Blocked | Blocked | Not readable |
| imgix.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Found |
| indeed.comConsumer and e-commerce | 0 of 12 | Allowed | Allowed | Allowed | Found |
| instacart.comTravel, jobs and marketplaces | 9 of 12 | Blocked | Blocked | Blocked | Not found |
| instagram.comSocial and community | 11 of 12 | Blocked | Blocked | Blocked | Not found (HTML page) |
| intel.comBig tech brands | 0 of 12 | Allowed | Allowed | Allowed | Found |
| intercom.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Found |
| investopedia.comFinance | not readable | n/a | n/a | n/a | Not readable |
| jasper.aiAI companies and tools | 0 of 12 | Allowed | Allowed | Allowed | Found |
| jetbrains.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| kayak.comTravel, jobs and marketplaces | 0 of 12 | Allowed | Allowed | Allowed | Found |
| khanacademy.orgEducation and reference | not readable | n/a | n/a | n/a | Not found (HTML page) |
| klaviyo.comSEO and marketing tools | 0 of 12 | Allowed | Allowed | Allowed | Found |
| kraken.comFinance | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| langchain.comAI companies and tools | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| lastpass.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| letterboxd.comEntertainment | 5 of 12 | Blocked | Blocked | Allowed | Not found |
| linear.appDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Found |
| linkedin.comConsumer and e-commerce | not readable | n/a | n/a | n/a | Not readable |
| llamaindex.aiAI companies and tools | 0 of 12 | Allowed | Allowed | Allowed | Found |
| loom.comProductivity | 1 of 12 | Blocked | Allowed | Allowed | Found |
| lovable.devAI companies and tools | 0 of 12 | Allowed | Allowed | Allowed | Found |
| lyft.comTravel, jobs and marketplaces | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| mailchimp.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Found |
| mastercard.comFinance | not readable | n/a | n/a | n/a | Not readable |
| masterclass.comEducation and reference | 0 of 12 | Allowed | Allowed | Allowed | Found |
| mayoclinic.orgHealth reference | not readable | n/a | n/a | n/a | Not readable |
| medium.comNews and media | not readable | n/a | n/a | n/a | Not readable |
| meilisearch.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Found |
| meta.comBig tech brands | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| microsoft.comEducation and reference | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| midjourney.comAI companies and tools | 4 of 12 | Blocked | Allowed | Allowed | Not found |
| miro.comProductivity | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| mistral.aiAI companies and tools | 0 of 12 | Allowed | Allowed | Allowed | Found |
| mit.eduEducation and reference | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| monday.comProductivity | 0 of 12 | Allowed | Allowed | Allowed | Found |
| mongodb.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Found |
| monster.comTravel, jobs and marketplaces | not readable | n/a | n/a | n/a | Not readable |
| moz.comSEO and marketing tools | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| mozilla.orgEducation and reference | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| mysql.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| nasa.govEducation and reference | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| neilpatel.comSEO and marketing tools | 0 of 12 | Allowed | Allowed | Allowed | Found |
| neon.techDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Found |
| nerdwallet.comFinance | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| netflix.comConsumer and e-commerce | 3 of 12 | Allowed | Allowed | Allowed | Not found (HTML page) |
| netlify.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Found |
| newrelic.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Found |
| nextjs.orgDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Found |
| nginx.comDeveloper tools and SaaS | not readable | n/a | n/a | n/a | Not found (HTML page) |
| nih.govHealth reference | not readable | n/a | n/a | n/a | Not readable |
| nike.comConsumer and e-commerce | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| nodejs.orgEducation and reference | 0 of 12 | Allowed | Allowed | Allowed | Found |
| notion.soDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Found |
| npmjs.comDeveloper tools and SaaS | not readable | n/a | n/a | n/a | Not readable |
| npr.orgNews and media | 10 of 12 | Blocked | Blocked | Blocked | Not found |
| nuxt.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Found |
| nvidia.comBig tech brands | 0 of 12 | Allowed | Allowed | Allowed | Found |
| nytimes.comNews and media | 11 of 12 | Blocked | Blocked | Blocked | Not found |
| obsidian.mdProductivity | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| okta.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Found |
| openai.comAI companies and tools | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| openrouter.aiAI companies and tools | 0 of 12 | Allowed | Allowed | Allowed | Found |
| oracle.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| pagerduty.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Found |
| paypal.comFinance | 0 of 12 | Allowed | Allowed | Allowed | Found |
| perplexity.aiAI companies and tools | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| php.netDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| pika.artAI companies and tools | 0 of 12 | Allowed | Allowed | Allowed | Found |
| pinecone.ioAI companies and tools | 0 of 12 | Allowed | Allowed | Allowed | Found |
| pinterest.comSocial and community | 11 of 12 | Blocked | Blocked | Blocked | Not found (HTML page) |
| planetscale.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Found |
| plausible.ioDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| postgresql.orgDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| posthog.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Found |
| producthunt.comStartup and review sites | 0 of 12 | Allowed | Allowed | Allowed | Found |
| pypi.orgDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| python.orgEducation and reference | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| quizlet.comEducation and reference | 5 of 12 | Blocked | Blocked | Blocked | Found |
| quora.comEducation and reference | 8 of 12 | Blocked | Blocked | Blocked | Not found |
| railway.appDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Found |
| reactjs.orgDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| realtor.comTravel, jobs and marketplaces | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| reddit.comEducation and reference | not readable | n/a | n/a | n/a | Not readable |
| redfin.comTravel, jobs and marketplaces | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| redhat.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Found |
| redis.ioDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Found |
| remix.runDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| render.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Found |
| replicate.comAI companies and tools | 0 of 12 | Allowed | Allowed | Allowed | Found |
| replit.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Found |
| reuters.comNews and media | 9 of 12 | Blocked | Blocked | Blocked | Not readable |
| revolut.comFinance | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| robinhood.comFinance | 0 of 12 | Allowed | Allowed | Allowed | Found |
| rottentomatoes.comEntertainment | not readable | n/a | n/a | n/a | Not readable |
| rubygems.orgDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| runwayml.comAI companies and tools | 0 of 12 | Allowed | Allowed | Allowed | Found |
| rust-lang.orgDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| salesforce.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Found |
| samsung.comBig tech brands | 0 of 12 | Allowed | Allowed | Allowed | Found |
| sanity.ioDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Found |
| schwab.comFinance | not readable | n/a | n/a | n/a | Not readable |
| screamingfrog.co.ukSEO and marketing tools | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| searchenginejournal.comSEO and marketing tools | 0 of 12 | Allowed | Allowed | Allowed | Found |
| searchengineland.comSEO and marketing tools | not readable | n/a | n/a | n/a | Not readable |
| semrush.comSEO and marketing tools | 0 of 12 | Allowed | Allowed | Allowed | Found |
| sendgrid.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Not found (HTML page) |
| sentry.ioDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Found |
| shopify.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Found |
| similarweb.comSEO and marketing tools | 0 of 12 | Allowed | Allowed | Allowed | Found |
| skillshare.comEducation and reference | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| slack.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Found |
| snapchat.comSocial and community | 5 of 12 | Blocked | Blocked | Allowed | Not found |
| snowflake.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| sony.comBig tech brands | not readable | n/a | n/a | n/a | Not readable |
| soundcloud.comEntertainment | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| splunk.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| spotify.comConsumer and e-commerce | 0 of 12 | Allowed | Allowed | Allowed | Not found (HTML page) |
| sproutsocial.comSEO and marketing tools | 0 of 12 | Allowed | Allowed | Allowed | Found |
| sqlite.orgDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| squarespace.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Found |
| squareup.comFinance | 0 of 12 | Allowed | Allowed | Allowed | Found |
| stability.aiAI companies and tools | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| stackoverflow.comEducation and reference | not readable | n/a | n/a | n/a | Not found |
| stanford.eduEducation and reference | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| strapi.ioDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| stripe.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Found |
| substack.comNews and media | 0 of 12 | Allowed | Allowed | Allowed | Found |
| suno.comAI companies and tools | 0 of 12 | Allowed | Allowed | Allowed | Found |
| supabase.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Found |
| surveymonkey.comProductivity | 0 of 12 | Allowed | Allowed | Allowed | Found |
| svelte.devDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Found |
| tailwindcss.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| target.comConsumer and e-commerce | 0 of 12 | Allowed | Allowed | Allowed | Found |
| techcrunch.comNews and media | 4 of 12 | Blocked | Blocked | Allowed | Not found |
| telegram.orgSocial and community | 0 of 12 | Allowed | Allowed | Allowed | Not found (HTML page) |
| tesla.comBig tech brands | not readable | n/a | n/a | n/a | Not readable |
| theguardian.comNews and media | 6 of 12 | Allowed | Blocked | Blocked | Not found |
| theverge.comNews and media | 9 of 12 | Allowed | Blocked | Blocked | Not found |
| tiktok.comSocial and community | 11 of 12 | Blocked | Blocked | Blocked | Not found (HTML page) |
| todoist.comProductivity | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| together.aiAI companies and tools | 1 of 12 | Allowed | Allowed | Allowed | Found |
| trello.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Not found (HTML page) |
| tripadvisor.comConsumer and e-commerce | 5 of 12 | Blocked | Blocked | Allowed | Not readable |
| trustpilot.comStartup and review sites | 3 of 12 | Allowed | Allowed | Allowed | Not readable |
| tumblr.comSocial and community | not readable | n/a | n/a | n/a | Not readable |
| twilio.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Found |
| twitch.tvSocial and community | 0 of 12 | Allowed | Allowed | Allowed | Not found (HTML page) |
| twitter.comSocial and community | 12 of 12 | Blocked | Blocked | Blocked | Not found |
| typeform.comProductivity | 0 of 12 | Allowed | Allowed | Allowed | Found |
| typesense.orgDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Found |
| uber.comConsumer and e-commerce | 0 of 12 | Allowed | Allowed | Allowed | Not readable |
| ubuntu.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Found |
| udemy.comEducation and reference | 0 of 12 | Allowed | Allowed | Allowed | Found |
| upstash.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Found |
| upwork.comTravel, jobs and marketplaces | 0 of 12 | Allowed | Allowed | Allowed | Found |
| v0.devAI companies and tools | 0 of 12 | Allowed | Allowed | Allowed | Not found (HTML page) |
| vercel.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Found |
| vimeo.comSocial and community | 1 of 12 | Allowed | Allowed | Allowed | Not found |
| visa.comFinance | not readable | n/a | n/a | n/a | Not found |
| vuejs.orgDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Found |
| w3schools.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| walmart.comConsumer and e-commerce | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| washingtonpost.comNews and media | 4 of 12 | Allowed | Blocked | Blocked | Not readable |
| weaviate.ioAI companies and tools | 0 of 12 | Allowed | Allowed | Allowed | Found |
| webflow.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Found |
| webmd.comHealth reference | 3 of 12 | Blocked | Blocked | Allowed | Not found |
| wellsfargo.comFinance | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| whatsapp.comSocial and community | 5 of 12 | Blocked | Blocked | Blocked | Not found |
| who.intEducation and reference | 1 of 12 | Allowed | Allowed | Allowed | Not found |
| wikipedia.orgNews and media | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| windsurf.comAI companies and tools | 0 of 12 | Allowed | Allowed | Allowed | Not found (HTML page) |
| wired.comNews and media | not readable | n/a | n/a | n/a | Not readable |
| wise.comFinance | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| wix.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Found |
| woocommerce.comConsumer and e-commerce | 0 of 12 | Allowed | Allowed | Allowed | Found |
| wordpress.comEducation and reference | 0 of 12 | Allowed | Allowed | Allowed | Found |
| wordpress.orgDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Found |
| x.aiAI companies and tools | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| x.comSocial and community | 12 of 12 | Blocked | Blocked | Blocked | Not found |
| ycombinator.comStartup and review sites | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| yelp.comConsumer and e-commerce | not readable | n/a | n/a | n/a | Not readable |
| yoast.comSEO and marketing tools | 0 of 12 | Allowed | Allowed | Allowed | Found |
| youtube.comSocial and community | 0 of 12 | Allowed | Allowed | Allowed | Not found |
| zendesk.comDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Found |
| zillow.comConsumer and e-commerce | 1 of 12 | Allowed | Allowed | Allowed | Not found |
| ziprecruiter.comTravel, jobs and marketplaces | 1 of 12 | Allowed | Allowed | Allowed | Found |
| zoom.usDeveloper tools and SaaS | 0 of 12 | Allowed | Allowed | Allowed | Not found |
Lists
Where does your site stand?
Free, no signup: checks your homepage, robots.txt, sitemap.xml and llms.txt and tells you what to fix first. For the per-crawler view use the AI Crawler Access Checker.
No signup and no payment details needed. Checks the homepage, robots.txt, sitemap.xml and llms.txt.
Method and limits
- Four plain GET requests per site on 19 November 2026:
/,/robots.txt,/llms.txt,/sitemap.xml. User agentseobolt-research/1.0; 6 sites that answered it with an error were read again with a standard browser user agent. - robots.txt is judged for the path
/with the RFC 9309 matching rules, the same code as the free checker. A site that only restricts sub-paths counts as allowed. - We test the 12 crawlers listed above, the ones whose operators document a robots.txt token (the same list as the free checker). We do not test others such as Bytespider, Meta-ExternalAgent or Amazonbot, so “allows all 12” does not mean every AI crawler is allowed.
- A missing robots.txt (404) means everything is allowed and is counted as such. A robots.txt we could not read (login wall, 403, timeout, HTML page) is excluded, not guessed.
llms.txtcounts as found only when the response is plain text, not an HTML page returned with status 200.- This is what each site publishes. It does not prove what a crawler actually does, and it cannot see firewall or CDN rules. Blocking or allowing a crawler is a business decision, not a defect.
- Snapshots are taken by hand, not continuously. Each page shows the date it was read.
Corrections and removal requests: support@seobolt.app. SEOBolt builds the audit tool this dataset is made with, so we have an interest in the topic.
Questions
Does blocking GPTBot remove a site from ChatGPT search?
No. OpenAI documents separate crawlers: GPTBot collects content that may be used to train its models, OAI-SearchBot is the one that surfaces sites in ChatGPT search results, and ChatGPT-User fetches a page when a user asks about it. A robots.txt that disallows GPTBot and allows OAI-SearchBot opts out of training and stays eligible for ChatGPT search.
Is robots.txt enforcement or a request?
A request. Crawlers that follow the Robots Exclusion Protocol honour it; nothing in the file stops a client that ignores it. Some vendors say their user-initiated fetchers (ChatGPT-User, Perplexity-User) may not apply robots.txt rules. A firewall or CDN can also block a crawler that robots.txt allows, which this dataset cannot see.
How was this data collected?
On 19 November 2026 we requested each site's homepage, /robots.txt, /llms.txt and /sitemap.xml with plain GET requests and judged robots.txt per bot with the RFC 9309 rules (a bot's own group beats the * group, the longest matching rule wins, Allow wins a tie), for the path /. 6 sites answered the research user agent with an error and were read again with a standard browser user agent.
A verdict about my site is wrong or I want it removed. What do I do?
Email support@seobolt.app with the domain. We will re-check it or remove the page and the CSV row, and keep it out of future snapshots.
How do I check my own site?
Use the free AI Crawler Access Checker or the free audit below. Both need no signup. They read your robots.txt per AI crawler and also send each crawler's published user agent to your homepage to see whether a firewall answers differently.