How to Make Your Site Visible to ChatGPT and Claude
TL;DR
- •Visibility starts with access. The crawlers that decide whether ChatGPT, Claude or Perplexity can show your pages are separate from the crawlers that gather training data. OpenAI, Anthropic and Perplexity each document them as independent robots.txt settings. Allow the search and user-fetch ones, then decide about training separately.
- •We re-cut our 100-site data set on 2026-11-18. 19 of 96 readable robots.txt files block at least one of six search or user-fetch bots (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User) from
/. None of the 38 sites that serve anllms.txtdo. No site blocked a bot from/llms.txtor/pricingwhile allowing/: blocks were whole-site. - •The checks are cheap. A copy-paste
curlscript is below. Run it before trusting any audit score, ours included: our audit reported a falsellms.txton 5 sites until PR #172, because a 200 can be your homepage.
We build SEOBolt, and the checks below are the ones its audit runs. We did not test whether any setting changes how often an AI product cites a site, and nothing here claims it does.
The three layers
- Can the bot reach the page? robots.txt rules and, separately, your firewall.
- Can it find the pages? A sitemap declared in robots.txt, and links in your HTML.
- Can it read them? Content in the raw HTML; an optional
llms.txtsits here.
Layer 1: which crawler is which
The roles below come from each vendor's page; the last column is from our 96 readable robots.txt files (path /).
| Bot | Vendor's description | Blocks `/` (of 96) |
|---|---|---|
| OAI-SearchBot | Surfaces sites in ChatGPT search features; opted-out sites "will not be shown in ChatGPT search answers" ([OpenAI](https://platform.openai.com/docs/bots)) | 10 |
| ChatGPT-User | Fetches pages when a user asks ChatGPT or a custom GPT; not used for automatic crawling (OpenAI) | 11 |
| GPTBot | Crawling for training OpenAI's models (OpenAI) | 20 |
| Claude-SearchBot | Indexes content to improve Claude's search results ([Anthropic](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)) | 15 |
| Claude-User | Fetches pages when a Claude user asks a question (Anthropic) | 14 |
| ClaudeBot | Collects content that could contribute to training (Anthropic) | 24 |
| PerplexityBot | Surfaces and links sites in Perplexity search; "not used to crawl content for AI foundation models" ([Perplexity](https://docs.perplexity.ai/guides/bots)) | 18 |
| Perplexity-User | Visits a page when a user asks a question; not for training (Perplexity) | 12 |
OpenAI says each setting is independent (allow OAI-SearchBot, disallow GPTBot) and that a change can take about 24 hours, so recheck after a day.
A minimal file that welcomes the search and user bots and opts out of training looks like this:
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /
User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xmlBlocking the training bots is a business decision, not an SEO one; make it on purpose. Robots.txt is a request, not a lock: RFC 9309 says its rules "are not a form of access authorization". The reverse is untested by us: a firewall that challenges unknown bots can lock out a crawler your robots.txt welcomes. Perplexity recommends allowing its published IP ranges as well as the user agent, so check any bot-protection layer too.
What the 100 sites do
| Cut | Result |
|---|---|
| Block at least one of the six search or user-fetch bots from `/` | 19 of 96 (20%) |
| Block ChatGPT's two bots (OAI-SearchBot or ChatGPT-User) | 11 of 96 |
| Blocked from `/llms.txt` or `/pricing` but not from `/` (any of the six) | 0 |
| Serve an `llms.txt` and block any of the eight bots above from `/` | 0 of 38 |
The zero in the third row means nobody in this sample carved out a path: a block is on or off for the whole site, so one fetch of /robots.txt tells you the state.
Layer 2: the sitemap
We fetched the first sitemap declared in robots.txt (live, 2026-11-18) for the 79 of 96 readable files that declare one:
| Declared sitemap | Sites |
|---|---|
| Valid XML at the first declared URL (`urlset` or `sitemapindex`) | 70 |
| Declared but not verified: bot-blocked response (401, 403 or no reply) 7, 404 1, gzip file we did not decode 1 | 9 |
| No `Sitemap:` line in a readable robots.txt | 17 (6 of them serve a valid `/sitemap.xml`) |
| robots.txt unreadable | 4 |
Three findings:
- •A root-only check misses sitemaps. 33 of the 79 declaring sites have no valid XML at
/sitemap.xml; 25 of those serve one at the declared URL. SEOBolt's audit accepts a robots.txt declaration for this reason. - •Declared does not mean working. trello.com declares a sitemap on a third-party domain that returned 404, and its own
/sitemap.xmlis an HTML page. theguardian.com and who.int declarehttp://URLs; notion.so declares URLs on notion.com. - •Many sites declare several. 37 of 79 list more than one, and 50 of the 70 valid first URLs are index files. We checked only the first URL, not what an index points to.
If you fix one thing: declare Sitemap: https://yourdomain/sitemap.xml as an absolute https:// URL and open it in a browser.
Layer 3: llms.txt, what counts
llms.txt is a Markdown site summary proposed at llmstxt.org. The vendors above do not document it as a requirement for their search products, and SEOBolt's audit copy says so. Treat it as optional, but if you add one, make sure it is the file.
PR #172 (merged 2026-11-17) fixed our audit's habit of trusting HTTP 200. An llms.txt now counts only if the response is not HTML, not empty, did not redirect to the site root, and has a Markdown heading, a Markdown link or a bulleted URL (src/lib/text-resource.ts). We ran that classifier over the bodies saved on 2026-11-17:
| Classifier result | Sites |
|---|---|
| ok (real text file) | 38 |
| missing (404, HTML page, empty, redirect to the root) | 53 |
| blocked (401, 403, 405 or no response, cannot tell) | 9 |
| malformed (text without heading or links) | 0 |
The 38 match the earlier count, and the 5 sites our old audit got wrong (sendgrid.com, trello.com, booking.com, spotify.com, netflix.com) are in the "missing" row. On 2026-11-18 sendgrid.com's /llms.txt still answered 200 with an HTML page after a redirect: the trap in one example.
What the 38 files look like, using the four context checks the SEOBolt audit applies (summary means a Markdown title plus a > line of 40 or more characters; the other three are links, features and pricing):
| Check in the file | Files (of 38) |
|---|---|
| H1 title | 35 |
| Title plus summary line | 33 |
| Absolute links | 38 |
| Mentions features, tools or workflow | 37 |
| Mentions pricing or a price | 33 |
| All four of summary, links, features, pricing | 28 |
Those checks are our heuristic for a product site, not a standard; a documentation site can have a good llms.txt without a price. Sizes run from 2.8 KB to 2.5 MB (twilio.com), median 12 KB. Our first post said "about 1 MB" for the maximum; that was wrong. Whether a model reads a 2.5 MB file is untested.
The check, as a script
This script tests all three layers for one domain. It needs curl and grep. I ran it on seobolt.app and sendgrid.com on 2026-11-18 (macOS).
#!/usr/bin/env bash
# Usage: ./ai-visibility-check.sh example.com
d="${1:?domain}"
UA="Mozilla/5.0 (compatible; my-check/1.0)"
echo "== robots.txt"
curl -sL -A "$UA" -o /tmp/robots.$$ -w "HTTP %{http_code} type %{content_type} final %{url_effective}\n" "https://$d/robots.txt"
echo "-- groups that name an AI crawler:"
grep -iE -A3 '^user-agent:[[:space:]]*(GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|Google-Extended|CCBot)' /tmp/robots.$$ || echo "(none named; the '*' group applies)"
echo "-- wildcard group:"; grep -iE -A4 '^user-agent:[[:space:]]*\*' /tmp/robots.$$ | head -8
echo "-- sitemap lines:"; grep -iE '^sitemap:' /tmp/robots.$$ || echo "(none)"
echo "== llms.txt"
curl -sL -A "$UA" -o /tmp/llms.$$ -w "HTTP %{http_code} type %{content_type} final %{url_effective} bytes %{size_download}\n" "https://$d/llms.txt"
head -c 300 /tmp/llms.$$; echo
echo "== sitemap.xml"
curl -sL -A "$UA" -o /tmp/sm.$$ -w "HTTP %{http_code} type %{content_type}\n" "https://$d/sitemap.xml"
grep -ocE '<(urlset|sitemapindex)' /tmp/sm.$$ | sed 's/^/xml root tags found: /'
rm -f /tmp/robots.$$ /tmp/llms.$$ /tmp/sm.$$Reading the output: a named group with Disallow: / blocks that bot, otherwise the * group decides. For llms.txt, a text/html type or a final URL that is not /llms.txt means no file; real ones start with # or a Markdown link. For the sitemap, xml root tags found: 1 or more means XML; 0 with HTTP 200 means a page, or a sitemap declared elsewhere.
It does not weigh competing rules (longest match wins, Allow beats Disallow on a tie, per RFC 9309) and cannot see a firewall. For precedence use a tester such as the free Robots.txt Tester. The free SEO audit runs these checks and the llms.txt Generator drafts a file; in the first study our audit could not read 27 of 100 homepages behind bot protection, so confirm anything it flags.
Methodology and data
- •Sample: the fixed list of 100 domains from our first study.
- •Robots.txt and llms.txt: the responses saved on 2026-11-17 (UTC). Blocking means a
Disallowwins for the path in the group that applies to the bot (its own group, else*; longest match; Allow wins ties), for the paths/,/llms.txtand/pricing. 96 of 100 robots.txt files were readable. Sub-path blocks other than these three were not examined. - •llms.txt classification: the
classifyTextResourcefunction from SEOBolt at commit962ffba(after PR #172), applied to the saved responses; no new request to those URLs was needed. The four content checks use the same regular expressions as the audit'sllms-txt-okrule. - •Sitemaps: fetched live on 2026-11-18, only the first declared URL per site, with the same user agent as before and one retry with a desktop Chrome user agent when the answer was not 200, 404 or 410. "Valid" means the body contains
<urlsetor<sitemapindex; a gzip sitemap was not decoded. - •Not tested: citation or traffic effects, WAF or CDN behaviour, whether models read
llms.txt, files that sitemap indexes point to. - •Correction to our first post: its maximum
llms.txtsize ("about 1 MB") should read 2.5 MB (twilio.com). - •Data and script: cut2.csv and cut2.mjs. The script reads the saved responses of the first study; the scripts to collect them are in its data folder.
- •Disclosure: we build SEOBolt.
Frequently asked questions
Should I block GPTBot and ClaudeBot?
That is a choice about training data, separate from letting the search and user-fetch bots through.
If I allow OAI-SearchBot, will ChatGPT cite me?
OpenAI says an opted-out site will not be shown in ChatGPT search answers; allowing guarantees nothing, and we did not measure citations.
Do I need an llms.txt?
The vendors we read do not document one as a requirement. If you add one, check that the URL returns a text file, not your homepage.
My robots.txt allows everything. Why would a bot still not see my site?
A WAF, CDN rule or login wall can stop a crawler robots.txt allows. We did not test any such setup.
Is the 100-site sample representative?
No: 100 hand-picked well-known domains, skewed toward tech, on one day.
Free tools for this: AI Visibility Checker, Robots.txt Tester, llms.txt Generator. No sign-up needed.
Check these issues on your site
Run a free homepage audit for technical SEO and AI-search blockers. No signup required.