Which top websites block ChatGPT, Claude and Perplexity?

We read the robots.txt, llms.txt and sitemap of 303 well-known websites on . Of the 271 whose robots.txt we could read, 34 block GPTBot (13%), 34 block ClaudeBot (13%) and 23 block PerplexityBot (8%). Only 12 block OpenAI's search crawler OAI-SearchBot (4%) against 34 for GPTBot: keeping out a training crawler and keeping out search are different choices.

Every site has its own page with the exact robots.txt rule behind each verdict. Download the CSV (free to reuse with a link back, CC BY 4.0).

Five kinds of robots.txt

226

Open to every AI crawler we test

No rule blocks any of the 12 crawlers on the homepage.

1

Blocks only crawlers that fetch a page on a user's request

ChatGPT-User, Claude-User or Perplexity-User is disallowed; the search and training crawlers are not.

21

Opted out of AI training, still open to AI search

Blocks training crawlers such as GPTBot, ClaudeBot or Google-Extended; ChatGPT, Claude and Perplexity search crawlers are allowed.

11

Blocks at least one AI search crawler

At least one of OAI-SearchBot, Claude-SearchBot or PerplexityBot is disallowed, so that assistant has less to cite.

12

Closed to AI search and training crawlers

All three search crawlers plus GPTBot and ClaudeBot are disallowed.

Of the 34 sites that block GPTBot, 28 name it in their own group and 6 only inherit a blanket User-agent: * disallow: for those, GPTBot is blocked because every crawler is, not because of an AI decision.

32 of 303 sites answered our robots.txt request with an error, a login wall or a non-robots page and are left out of every percentage on this page.

Crawler by crawler

Number of sites whose robots.txt blocks each AI crawler on the homepage
CrawlerWhat it doesSites blocking it
CCBotCommon CrawlModel training37 of 271 (14%)
GPTBotOpenAIModel training34 of 271 (13%)
ClaudeBotAnthropicModel training34 of 271 (13%)
Applebot-ExtendedAppleModel training34 of 271 (13%)
Google-ExtendedGoogleModel training24 of 271 (9%)
PerplexityBotPerplexitySearch23 of 271 (8%)
Claude-SearchBotAnthropicSearch17 of 271 (6%)
Claude-UserAnthropicUser request17 of 271 (6%)
Perplexity-UserPerplexityUser request17 of 271 (6%)
ChatGPT-UserOpenAIUser request13 of 271 (5%)
OAI-SearchBotOpenAISearch12 of 271 (4%)
ApplebotAppleSearch3 of 271 (1%)

Operator, purpose and user-agent tokens come from each vendor's own documentation; the sources are listed on the AI Crawler Access Checker.

By kind of site

Share of sites blocking GPTBot and ClaudeBot, by category
CategorySites readBlock GPTBotBlock ClaudeBotBlock any AI search crawlerPublish llms.txt
Social and community138 (62%)8 (62%)7 (54%)0 of 13 (0%)
News and media148 (57%)11 (79%)10 (71%)2 of 11 (18%)
Consumer and e-commerce205 (25%)5 (25%)2 (10%)7 of 18 (39%)
Startup and review sites51 (20%)1 (20%)0 (0%)2 of 4 (50%)
Travel, jobs and marketplaces101 (10%)1 (10%)1 (10%)4 of 10 (40%)
Education and reference222 (9%)2 (9%)2 (9%)8 of 22 (36%)
Productivity131 (8%)0 (0%)0 (0%)7 of 13 (54%)
AI companies and tools301 (3%)0 (0%)0 (0%)15 of 30 (50%)
Developer tools and SaaS1022 (2%)2 (2%)0 (0%)61 of 101 (60%)
Big tech brands50 (0%)0 (0%)0 (0%)3 of 5 (60%)
Finance140 (0%)0 (0%)0 (0%)4 of 12 (33%)
SEO and marketing tools160 (0%)0 (0%)0 (0%)10 of 16 (63%)

Categories with fewer than five readable sites are not shown; llms.txt shares count only sites whose /llms.txt we could read. Our sample is popular, mostly English-language sites, not a random sample of the web.

All 303 sites

Showing 303 of 303 sites.

AI crawler access by website, from each site's robots.txt
SiteBlockedGPTBotClaudeBotPerplexityBotllms.txt
1password.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedNot found
activecampaign.comSEO and marketing tools0 of 12AllowedAllowedAllowedNot found
adidas.comConsumer and e-commercenot readablen/an/an/aNot readable
ahrefs.comSEO and marketing tools0 of 12AllowedAllowedAllowedNot found
airbnb.comConsumer and e-commerce3 of 12BlockedBlockedAllowedNot found
airtable.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedNot found
algolia.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedFound
amazon.comConsumer and e-commerce10 of 12BlockedBlockedBlockedNot found
amd.comBig tech brands0 of 12AllowedAllowedAllowedNot found
americanexpress.comFinance0 of 12AllowedAllowedAllowedNot found
angular.ioDeveloper tools and SaaS0 of 12AllowedAllowedAllowedNot found
anthropic.comAI companies and tools0 of 12AllowedAllowedAllowedNot found
apnews.comNews and media8 of 12BlockedBlockedBlockedNot found
apple.comEducation and reference0 of 12AllowedAllowedAllowedNot found
arstechnica.comNews and medianot readablen/an/an/aNot readable
asana.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedFound
astro.buildDeveloper tools and SaaS0 of 12AllowedAllowedAllowedNot found
atlassian.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedFound
auth0.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedFound
aws.amazon.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedNot found
azure.microsoft.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedNot found
backlinko.comSEO and marketing tools0 of 12AllowedAllowedAllowedNot found
bankofamerica.comFinance0 of 12AllowedAllowedAllowedNot found
basecamp.comProductivity0 of 12AllowedAllowedAllowedNot found
bbc.comNews and medianot readablen/an/an/aNot readable
beehiiv.comSEO and marketing tools0 of 12AllowedAllowedAllowedFound
bestbuy.comConsumer and e-commerce0 of 12AllowedAllowedAllowedNot found
bigcommerce.comConsumer and e-commerce0 of 12AllowedAllowedAllowedFound
binance.comFinance0 of 12AllowedAllowedAllowedNot readable
bitwarden.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedFound
bloomberg.comNews and media6 of 12BlockedBlockedBlockedNot readable
bolt.newAI companies and tools0 of 12AllowedAllowedAllowedNot found (HTML page)
booking.comConsumer and e-commerce0 of 12AllowedAllowedAllowedNot found
brevo.comSEO and marketing tools0 of 12AllowedAllowedAllowedNot found
brilliant.orgEducation and reference0 of 12AllowedAllowedAllowedFound
buffer.comSEO and marketing tools0 of 12AllowedAllowedAllowedFound
calendly.comProductivity1 of 12AllowedAllowedAllowedFound
canva.comDeveloper tools and SaaS4 of 12BlockedBlockedAllowedNot found
capterra.comStartup and review sitesnot readablen/an/an/aNot readable
cdc.govHealth reference0 of 12AllowedAllowedAllowedNot found
character.aiAI companies and tools0 of 12AllowedAllowedAllowedNot found
chase.comFinance0 of 12AllowedAllowedAllowedNot found
chegg.comEducation and reference0 of 12AllowedAllowedAllowedNot found
clerk.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedFound
clickup.comProductivity0 of 12AllowedAllowedAllowedFound
cloud.google.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedNot found (HTML page)
cloudflare.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedFound
cloudinary.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedFound
cnn.comNews and media10 of 12BlockedBlockedBlockedNot found
cockroachlabs.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedNot found
codepen.ioDeveloper tools and SaaSnot readablen/an/an/aNot readable
codesandbox.ioDeveloper tools and SaaSnot readablen/an/an/aNot readable
cohere.comAI companies and tools0 of 12AllowedAllowedAllowedFound
coinbase.comFinance0 of 12AllowedAllowedAllowedFound
confluent.ioDeveloper tools and SaaS0 of 12AllowedAllowedAllowedNot found
contentful.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedNot found
convertkit.comSEO and marketing toolsnot readablen/an/an/aNot readable
copy.aiAI companies and tools0 of 12AllowedAllowedAllowedNot found
coursera.orgEducation and reference0 of 12AllowedAllowedAllowedFound
crates.ioDeveloper tools and SaaS0 of 12AllowedAllowedAllowedNot found
crunchbase.comStartup and review sites5 of 12BlockedBlockedAllowedNot found
cursor.comAI companies and tools0 of 12AllowedAllowedAllowedFound
databricks.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedFound
datadoghq.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedFound
deepl.comProductivity0 of 12AllowedAllowedAllowedNot found
deepseek.comAI companies and tools0 of 12AllowedAllowedAllowedNot found
dev.toNews and media0 of 12AllowedAllowedAllowedFound
developer.mozilla.orgDeveloper tools and SaaS0 of 12AllowedAllowedAllowedNot found
digitalocean.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedFound
discord.comSocial and community0 of 12AllowedAllowedAllowedNot found
disneyplus.comEntertainmentnot readablen/an/an/aNot found (HTML page)
docker.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedNot found
doordash.comTravel, jobs and marketplaces0 of 12AllowedAllowedAllowedNot found
dropbox.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedFound
duolingo.comEducation and reference0 of 12AllowedAllowedAllowedNot found (HTML page)
ebay.comConsumer and e-commerce5 of 12BlockedBlockedBlockedNot found
edx.orgEducation and reference0 of 12AllowedAllowedAllowedFound
elastic.coDeveloper tools and SaaS0 of 12AllowedAllowedAllowedFound
elevenlabs.ioAI companies and tools0 of 12AllowedAllowedAllowedFound
etsy.comConsumer and e-commerce0 of 12AllowedAllowedAllowedFound
evernote.comProductivity0 of 12AllowedAllowedAllowedFound
expedia.comConsumer and e-commerce0 of 12AllowedAllowedAllowedFound
facebook.comSocial and community11 of 12BlockedBlockedBlockedNot found (HTML page)
fidelity.comFinance0 of 12AllowedAllowedAllowedNot readable
figma.comDeveloper tools and SaaS4 of 12BlockedBlockedAllowedNot found
fiverr.comTravel, jobs and marketplaces0 of 12AllowedAllowedAllowedFound
fly.ioDeveloper tools and SaaS0 of 12AllowedAllowedAllowedFound
forbes.comNews and media5 of 12BlockedBlockedBlockedNot found
framer.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedFound
freecodecamp.orgDeveloper tools and SaaS0 of 12AllowedAllowedAllowedNot found
freelancer.comTravel, jobs and marketplaces0 of 12AllowedAllowedAllowedNot found
g2.comStartup and review sites0 of 12AllowedAllowedAllowedFound
getbootstrap.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedFound
ghost.orgDeveloper tools and SaaS0 of 12AllowedAllowedAllowedNot found
github.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedFound
gitlab.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedNot readable
glassdoor.comConsumer and e-commerce5 of 12BlockedBlockedAllowedFound
glitch.comDeveloper tools and SaaSnot readablen/an/an/aNot found (HTML page)
golang.orgDeveloper tools and SaaS0 of 12AllowedAllowedAllowedNot found
goodreads.comEntertainment2 of 12BlockedAllowedAllowedNot found
google.comEducation and reference0 of 12AllowedAllowedAllowedNot found
grafana.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedFound
grammarly.comProductivity0 of 12AllowedAllowedAllowedNot found
groq.comAI companies and tools0 of 12AllowedAllowedAllowedNot found
harvard.eduEducation and reference0 of 12AllowedAllowedAllowedNot found
hashnode.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedFound
healthline.comHealth reference4 of 12BlockedBlockedAllowedNot found
heroku.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedFound
homedepot.comConsumer and e-commercenot readablen/an/an/aNot readable
hootsuite.comSEO and marketing tools0 of 12AllowedAllowedAllowedFound
hubspot.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedFound
huggingface.coAI companies and tools0 of 12AllowedAllowedAllowedNot found
hulu.comEntertainmentnot readablen/an/an/aNot found (HTML page)
ibm.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedNot found
ikea.comConsumer and e-commerce0 of 12AllowedAllowedAllowedNot found
imdb.comEntertainment11 of 12BlockedBlockedBlockedNot readable
imgix.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedFound
indeed.comConsumer and e-commerce0 of 12AllowedAllowedAllowedFound
instacart.comTravel, jobs and marketplaces9 of 12BlockedBlockedBlockedNot found
instagram.comSocial and community11 of 12BlockedBlockedBlockedNot found (HTML page)
intel.comBig tech brands0 of 12AllowedAllowedAllowedFound
intercom.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedFound
investopedia.comFinancenot readablen/an/an/aNot readable
jasper.aiAI companies and tools0 of 12AllowedAllowedAllowedFound
jetbrains.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedNot found
kayak.comTravel, jobs and marketplaces0 of 12AllowedAllowedAllowedFound
khanacademy.orgEducation and referencenot readablen/an/an/aNot found (HTML page)
klaviyo.comSEO and marketing tools0 of 12AllowedAllowedAllowedFound
kraken.comFinance0 of 12AllowedAllowedAllowedNot found
langchain.comAI companies and tools0 of 12AllowedAllowedAllowedNot found
lastpass.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedNot found
letterboxd.comEntertainment5 of 12BlockedBlockedAllowedNot found
linear.appDeveloper tools and SaaS0 of 12AllowedAllowedAllowedFound
linkedin.comConsumer and e-commercenot readablen/an/an/aNot readable
llamaindex.aiAI companies and tools0 of 12AllowedAllowedAllowedFound
loom.comProductivity1 of 12BlockedAllowedAllowedFound
lovable.devAI companies and tools0 of 12AllowedAllowedAllowedFound
lyft.comTravel, jobs and marketplaces0 of 12AllowedAllowedAllowedNot found
mailchimp.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedFound
mastercard.comFinancenot readablen/an/an/aNot readable
masterclass.comEducation and reference0 of 12AllowedAllowedAllowedFound
mayoclinic.orgHealth referencenot readablen/an/an/aNot readable
medium.comNews and medianot readablen/an/an/aNot readable
meilisearch.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedFound
meta.comBig tech brands0 of 12AllowedAllowedAllowedNot found
microsoft.comEducation and reference0 of 12AllowedAllowedAllowedNot found
midjourney.comAI companies and tools4 of 12BlockedAllowedAllowedNot found
miro.comProductivity0 of 12AllowedAllowedAllowedNot found
mistral.aiAI companies and tools0 of 12AllowedAllowedAllowedFound
mit.eduEducation and reference0 of 12AllowedAllowedAllowedNot found
monday.comProductivity0 of 12AllowedAllowedAllowedFound
mongodb.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedFound
monster.comTravel, jobs and marketplacesnot readablen/an/an/aNot readable
moz.comSEO and marketing tools0 of 12AllowedAllowedAllowedNot found
mozilla.orgEducation and reference0 of 12AllowedAllowedAllowedNot found
mysql.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedNot found
nasa.govEducation and reference0 of 12AllowedAllowedAllowedNot found
neilpatel.comSEO and marketing tools0 of 12AllowedAllowedAllowedFound
neon.techDeveloper tools and SaaS0 of 12AllowedAllowedAllowedFound
nerdwallet.comFinance0 of 12AllowedAllowedAllowedNot found
netflix.comConsumer and e-commerce3 of 12AllowedAllowedAllowedNot found (HTML page)
netlify.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedFound
newrelic.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedFound
nextjs.orgDeveloper tools and SaaS0 of 12AllowedAllowedAllowedFound
nginx.comDeveloper tools and SaaSnot readablen/an/an/aNot found (HTML page)
nih.govHealth referencenot readablen/an/an/aNot readable
nike.comConsumer and e-commerce0 of 12AllowedAllowedAllowedNot found
nodejs.orgEducation and reference0 of 12AllowedAllowedAllowedFound
notion.soDeveloper tools and SaaS0 of 12AllowedAllowedAllowedFound
npmjs.comDeveloper tools and SaaSnot readablen/an/an/aNot readable
npr.orgNews and media10 of 12BlockedBlockedBlockedNot found
nuxt.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedFound
nvidia.comBig tech brands0 of 12AllowedAllowedAllowedFound
nytimes.comNews and media11 of 12BlockedBlockedBlockedNot found
obsidian.mdProductivity0 of 12AllowedAllowedAllowedNot found
okta.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedFound
openai.comAI companies and tools0 of 12AllowedAllowedAllowedNot found
openrouter.aiAI companies and tools0 of 12AllowedAllowedAllowedFound
oracle.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedNot found
pagerduty.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedFound
paypal.comFinance0 of 12AllowedAllowedAllowedFound
perplexity.aiAI companies and tools0 of 12AllowedAllowedAllowedNot found
php.netDeveloper tools and SaaS0 of 12AllowedAllowedAllowedNot found
pika.artAI companies and tools0 of 12AllowedAllowedAllowedFound
pinecone.ioAI companies and tools0 of 12AllowedAllowedAllowedFound
pinterest.comSocial and community11 of 12BlockedBlockedBlockedNot found (HTML page)
planetscale.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedFound
plausible.ioDeveloper tools and SaaS0 of 12AllowedAllowedAllowedNot found
postgresql.orgDeveloper tools and SaaS0 of 12AllowedAllowedAllowedNot found
posthog.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedFound
producthunt.comStartup and review sites0 of 12AllowedAllowedAllowedFound
pypi.orgDeveloper tools and SaaS0 of 12AllowedAllowedAllowedNot found
python.orgEducation and reference0 of 12AllowedAllowedAllowedNot found
quizlet.comEducation and reference5 of 12BlockedBlockedBlockedFound
quora.comEducation and reference8 of 12BlockedBlockedBlockedNot found
railway.appDeveloper tools and SaaS0 of 12AllowedAllowedAllowedFound
reactjs.orgDeveloper tools and SaaS0 of 12AllowedAllowedAllowedNot found
realtor.comTravel, jobs and marketplaces0 of 12AllowedAllowedAllowedNot found
reddit.comEducation and referencenot readablen/an/an/aNot readable
redfin.comTravel, jobs and marketplaces0 of 12AllowedAllowedAllowedNot found
redhat.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedFound
redis.ioDeveloper tools and SaaS0 of 12AllowedAllowedAllowedFound
remix.runDeveloper tools and SaaS0 of 12AllowedAllowedAllowedNot found
render.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedFound
replicate.comAI companies and tools0 of 12AllowedAllowedAllowedFound
replit.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedFound
reuters.comNews and media9 of 12BlockedBlockedBlockedNot readable
revolut.comFinance0 of 12AllowedAllowedAllowedNot found
robinhood.comFinance0 of 12AllowedAllowedAllowedFound
rottentomatoes.comEntertainmentnot readablen/an/an/aNot readable
rubygems.orgDeveloper tools and SaaS0 of 12AllowedAllowedAllowedNot found
runwayml.comAI companies and tools0 of 12AllowedAllowedAllowedFound
rust-lang.orgDeveloper tools and SaaS0 of 12AllowedAllowedAllowedNot found
salesforce.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedFound
samsung.comBig tech brands0 of 12AllowedAllowedAllowedFound
sanity.ioDeveloper tools and SaaS0 of 12AllowedAllowedAllowedFound
schwab.comFinancenot readablen/an/an/aNot readable
screamingfrog.co.ukSEO and marketing tools0 of 12AllowedAllowedAllowedNot found
searchenginejournal.comSEO and marketing tools0 of 12AllowedAllowedAllowedFound
searchengineland.comSEO and marketing toolsnot readablen/an/an/aNot readable
semrush.comSEO and marketing tools0 of 12AllowedAllowedAllowedFound
sendgrid.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedNot found (HTML page)
sentry.ioDeveloper tools and SaaS0 of 12AllowedAllowedAllowedFound
shopify.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedFound
similarweb.comSEO and marketing tools0 of 12AllowedAllowedAllowedFound
skillshare.comEducation and reference0 of 12AllowedAllowedAllowedNot found
slack.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedFound
snapchat.comSocial and community5 of 12BlockedBlockedAllowedNot found
snowflake.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedNot found
sony.comBig tech brandsnot readablen/an/an/aNot readable
soundcloud.comEntertainment0 of 12AllowedAllowedAllowedNot found
splunk.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedNot found
spotify.comConsumer and e-commerce0 of 12AllowedAllowedAllowedNot found (HTML page)
sproutsocial.comSEO and marketing tools0 of 12AllowedAllowedAllowedFound
sqlite.orgDeveloper tools and SaaS0 of 12AllowedAllowedAllowedNot found
squarespace.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedFound
squareup.comFinance0 of 12AllowedAllowedAllowedFound
stability.aiAI companies and tools0 of 12AllowedAllowedAllowedNot found
stackoverflow.comEducation and referencenot readablen/an/an/aNot found
stanford.eduEducation and reference0 of 12AllowedAllowedAllowedNot found
strapi.ioDeveloper tools and SaaS0 of 12AllowedAllowedAllowedNot found
stripe.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedFound
substack.comNews and media0 of 12AllowedAllowedAllowedFound
suno.comAI companies and tools0 of 12AllowedAllowedAllowedFound
supabase.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedFound
surveymonkey.comProductivity0 of 12AllowedAllowedAllowedFound
svelte.devDeveloper tools and SaaS0 of 12AllowedAllowedAllowedFound
tailwindcss.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedNot found
target.comConsumer and e-commerce0 of 12AllowedAllowedAllowedFound
techcrunch.comNews and media4 of 12BlockedBlockedAllowedNot found
telegram.orgSocial and community0 of 12AllowedAllowedAllowedNot found (HTML page)
tesla.comBig tech brandsnot readablen/an/an/aNot readable
theguardian.comNews and media6 of 12AllowedBlockedBlockedNot found
theverge.comNews and media9 of 12AllowedBlockedBlockedNot found
tiktok.comSocial and community11 of 12BlockedBlockedBlockedNot found (HTML page)
todoist.comProductivity0 of 12AllowedAllowedAllowedNot found
together.aiAI companies and tools1 of 12AllowedAllowedAllowedFound
trello.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedNot found (HTML page)
tripadvisor.comConsumer and e-commerce5 of 12BlockedBlockedAllowedNot readable
trustpilot.comStartup and review sites3 of 12AllowedAllowedAllowedNot readable
tumblr.comSocial and communitynot readablen/an/an/aNot readable
twilio.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedFound
twitch.tvSocial and community0 of 12AllowedAllowedAllowedNot found (HTML page)
twitter.comSocial and community12 of 12BlockedBlockedBlockedNot found
typeform.comProductivity0 of 12AllowedAllowedAllowedFound
typesense.orgDeveloper tools and SaaS0 of 12AllowedAllowedAllowedFound
uber.comConsumer and e-commerce0 of 12AllowedAllowedAllowedNot readable
ubuntu.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedFound
udemy.comEducation and reference0 of 12AllowedAllowedAllowedFound
upstash.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedFound
upwork.comTravel, jobs and marketplaces0 of 12AllowedAllowedAllowedFound
v0.devAI companies and tools0 of 12AllowedAllowedAllowedNot found (HTML page)
vercel.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedFound
vimeo.comSocial and community1 of 12AllowedAllowedAllowedNot found
visa.comFinancenot readablen/an/an/aNot found
vuejs.orgDeveloper tools and SaaS0 of 12AllowedAllowedAllowedFound
w3schools.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedNot found
walmart.comConsumer and e-commerce0 of 12AllowedAllowedAllowedNot found
washingtonpost.comNews and media4 of 12AllowedBlockedBlockedNot readable
weaviate.ioAI companies and tools0 of 12AllowedAllowedAllowedFound
webflow.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedFound
webmd.comHealth reference3 of 12BlockedBlockedAllowedNot found
wellsfargo.comFinance0 of 12AllowedAllowedAllowedNot found
whatsapp.comSocial and community5 of 12BlockedBlockedBlockedNot found
who.intEducation and reference1 of 12AllowedAllowedAllowedNot found
wikipedia.orgNews and media0 of 12AllowedAllowedAllowedNot found
windsurf.comAI companies and tools0 of 12AllowedAllowedAllowedNot found (HTML page)
wired.comNews and medianot readablen/an/an/aNot readable
wise.comFinance0 of 12AllowedAllowedAllowedNot found
wix.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedFound
woocommerce.comConsumer and e-commerce0 of 12AllowedAllowedAllowedFound
wordpress.comEducation and reference0 of 12AllowedAllowedAllowedFound
wordpress.orgDeveloper tools and SaaS0 of 12AllowedAllowedAllowedFound
x.aiAI companies and tools0 of 12AllowedAllowedAllowedNot found
x.comSocial and community12 of 12BlockedBlockedBlockedNot found
ycombinator.comStartup and review sites0 of 12AllowedAllowedAllowedNot found
yelp.comConsumer and e-commercenot readablen/an/an/aNot readable
yoast.comSEO and marketing tools0 of 12AllowedAllowedAllowedFound
youtube.comSocial and community0 of 12AllowedAllowedAllowedNot found
zendesk.comDeveloper tools and SaaS0 of 12AllowedAllowedAllowedFound
zillow.comConsumer and e-commerce1 of 12AllowedAllowedAllowedNot found
ziprecruiter.comTravel, jobs and marketplaces1 of 12AllowedAllowedAllowedFound
zoom.usDeveloper tools and SaaS0 of 12AllowedAllowedAllowedNot found

Lists

Where does your site stand?

Free, no signup: checks your homepage, robots.txt, sitemap.xml and llms.txt and tells you what to fix first. For the per-crawler view use the AI Crawler Access Checker.

No signup and no payment details needed. Checks the homepage, robots.txt, sitemap.xml and llms.txt.

Method and limits

  • Four plain GET requests per site on 19 November 2026: /, /robots.txt, /llms.txt, /sitemap.xml. User agent seobolt-research/1.0; 6 sites that answered it with an error were read again with a standard browser user agent.
  • robots.txt is judged for the path / with the RFC 9309 matching rules, the same code as the free checker. A site that only restricts sub-paths counts as allowed.
  • We test the 12 crawlers listed above, the ones whose operators document a robots.txt token (the same list as the free checker). We do not test others such as Bytespider, Meta-ExternalAgent or Amazonbot, so “allows all 12” does not mean every AI crawler is allowed.
  • A missing robots.txt (404) means everything is allowed and is counted as such. A robots.txt we could not read (login wall, 403, timeout, HTML page) is excluded, not guessed.
  • llms.txt counts as found only when the response is plain text, not an HTML page returned with status 200.
  • This is what each site publishes. It does not prove what a crawler actually does, and it cannot see firewall or CDN rules. Blocking or allowing a crawler is a business decision, not a defect.
  • Snapshots are taken by hand, not continuously. Each page shows the date it was read.

Corrections and removal requests: support@seobolt.app. SEOBolt builds the audit tool this dataset is made with, so we have an interest in the topic.

Questions

Does blocking GPTBot remove a site from ChatGPT search?

No. OpenAI documents separate crawlers: GPTBot collects content that may be used to train its models, OAI-SearchBot is the one that surfaces sites in ChatGPT search results, and ChatGPT-User fetches a page when a user asks about it. A robots.txt that disallows GPTBot and allows OAI-SearchBot opts out of training and stays eligible for ChatGPT search.

Is robots.txt enforcement or a request?

A request. Crawlers that follow the Robots Exclusion Protocol honour it; nothing in the file stops a client that ignores it. Some vendors say their user-initiated fetchers (ChatGPT-User, Perplexity-User) may not apply robots.txt rules. A firewall or CDN can also block a crawler that robots.txt allows, which this dataset cannot see.

How was this data collected?

On 19 November 2026 we requested each site's homepage, /robots.txt, /llms.txt and /sitemap.xml with plain GET requests and judged robots.txt per bot with the RFC 9309 rules (a bot's own group beats the * group, the longest matching rule wins, Allow wins a tie), for the path /. 6 sites answered the research user agent with an error and were read again with a standard browser user agent.

A verdict about my site is wrong or I want it removed. What do I do?

Email support@seobolt.app with the domain. We will re-check it or remove the page and the CSV row, and keep it out of future snapshots.

How do I check my own site?

Use the free AI Crawler Access Checker or the free audit below. Both need no signup. They read your robots.txt per AI crawler and also send each crawler's published user agent to your homepage to see whether a firewall answers differently.