Back to Blog

How to Make Your Site Visible to ChatGPT and Claude

2026-11-189 min

TL;DR

  • •Visibility starts with access. The crawlers that decide whether ChatGPT, Claude or Perplexity can show your pages are separate from the crawlers that gather training data. OpenAI, Anthropic and Perplexity each document them as independent robots.txt settings. Allow the search and user-fetch ones, then decide about training separately.
  • •We re-cut our 100-site data set on 2026-11-18. 19 of 96 readable robots.txt files block at least one of six search or user-fetch bots (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User) from /. None of the 38 sites that serve an llms.txt do. No site blocked a bot from /llms.txt or /pricing while allowing /: blocks were whole-site.
  • •The checks are cheap. A copy-paste curl script is below. Run it before trusting any audit score, ours included: our audit reported a false llms.txt on 5 sites until PR #172, because a 200 can be your homepage.

We build SEOBolt, and the checks below are the ones its audit runs. We did not test whether any setting changes how often an AI product cites a site, and nothing here claims it does.

The three layers

  1. Can the bot reach the page? robots.txt rules and, separately, your firewall.
  2. Can it find the pages? A sitemap declared in robots.txt, and links in your HTML.
  3. Can it read them? Content in the raw HTML; an optional llms.txt sits here.

Layer 1: which crawler is which

The roles below come from each vendor's page; the last column is from our 96 readable robots.txt files (path /).

BotVendor's descriptionBlocks `/` (of 96)
OAI-SearchBotSurfaces sites in ChatGPT search features; opted-out sites "will not be shown in ChatGPT search answers" ([OpenAI](https://platform.openai.com/docs/bots))10
ChatGPT-UserFetches pages when a user asks ChatGPT or a custom GPT; not used for automatic crawling (OpenAI)11
GPTBotCrawling for training OpenAI's models (OpenAI)20
Claude-SearchBotIndexes content to improve Claude's search results ([Anthropic](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler))15
Claude-UserFetches pages when a Claude user asks a question (Anthropic)14
ClaudeBotCollects content that could contribute to training (Anthropic)24
PerplexityBotSurfaces and links sites in Perplexity search; "not used to crawl content for AI foundation models" ([Perplexity](https://docs.perplexity.ai/guides/bots))18
Perplexity-UserVisits a page when a user asks a question; not for training (Perplexity)12

OpenAI says each setting is independent (allow OAI-SearchBot, disallow GPTBot) and that a change can take about 24 hours, so recheck after a day.

A minimal file that welcomes the search and user bots and opts out of training looks like this:

User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /

User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Blocking the training bots is a business decision, not an SEO one; make it on purpose. Robots.txt is a request, not a lock: RFC 9309 says its rules "are not a form of access authorization". The reverse is untested by us: a firewall that challenges unknown bots can lock out a crawler your robots.txt welcomes. Perplexity recommends allowing its published IP ranges as well as the user agent, so check any bot-protection layer too.

What the 100 sites do

CutResult
Block at least one of the six search or user-fetch bots from `/`19 of 96 (20%)
Block ChatGPT's two bots (OAI-SearchBot or ChatGPT-User)11 of 96
Blocked from `/llms.txt` or `/pricing` but not from `/` (any of the six)0
Serve an `llms.txt` and block any of the eight bots above from `/`0 of 38

The zero in the third row means nobody in this sample carved out a path: a block is on or off for the whole site, so one fetch of /robots.txt tells you the state.

Layer 2: the sitemap

We fetched the first sitemap declared in robots.txt (live, 2026-11-18) for the 79 of 96 readable files that declare one:

Declared sitemapSites
Valid XML at the first declared URL (`urlset` or `sitemapindex`)70
Declared but not verified: bot-blocked response (401, 403 or no reply) 7, 404 1, gzip file we did not decode 19
No `Sitemap:` line in a readable robots.txt17 (6 of them serve a valid `/sitemap.xml`)
robots.txt unreadable4

Three findings:

  • •A root-only check misses sitemaps. 33 of the 79 declaring sites have no valid XML at /sitemap.xml; 25 of those serve one at the declared URL. SEOBolt's audit accepts a robots.txt declaration for this reason.
  • •Declared does not mean working. trello.com declares a sitemap on a third-party domain that returned 404, and its own /sitemap.xml is an HTML page. theguardian.com and who.int declare http:// URLs; notion.so declares URLs on notion.com.
  • •Many sites declare several. 37 of 79 list more than one, and 50 of the 70 valid first URLs are index files. We checked only the first URL, not what an index points to.

If you fix one thing: declare Sitemap: https://yourdomain/sitemap.xml as an absolute https:// URL and open it in a browser.

Layer 3: llms.txt, what counts

llms.txt is a Markdown site summary proposed at llmstxt.org. The vendors above do not document it as a requirement for their search products, and SEOBolt's audit copy says so. Treat it as optional, but if you add one, make sure it is the file.

PR #172 (merged 2026-11-17) fixed our audit's habit of trusting HTTP 200. An llms.txt now counts only if the response is not HTML, not empty, did not redirect to the site root, and has a Markdown heading, a Markdown link or a bulleted URL (src/lib/text-resource.ts). We ran that classifier over the bodies saved on 2026-11-17:

Classifier resultSites
ok (real text file)38
missing (404, HTML page, empty, redirect to the root)53
blocked (401, 403, 405 or no response, cannot tell)9
malformed (text without heading or links)0

The 38 match the earlier count, and the 5 sites our old audit got wrong (sendgrid.com, trello.com, booking.com, spotify.com, netflix.com) are in the "missing" row. On 2026-11-18 sendgrid.com's /llms.txt still answered 200 with an HTML page after a redirect: the trap in one example.

What the 38 files look like, using the four context checks the SEOBolt audit applies (summary means a Markdown title plus a > line of 40 or more characters; the other three are links, features and pricing):

Check in the fileFiles (of 38)
H1 title35
Title plus summary line33
Absolute links38
Mentions features, tools or workflow37
Mentions pricing or a price33
All four of summary, links, features, pricing28

Those checks are our heuristic for a product site, not a standard; a documentation site can have a good llms.txt without a price. Sizes run from 2.8 KB to 2.5 MB (twilio.com), median 12 KB. Our first post said "about 1 MB" for the maximum; that was wrong. Whether a model reads a 2.5 MB file is untested.

The check, as a script

This script tests all three layers for one domain. It needs curl and grep. I ran it on seobolt.app and sendgrid.com on 2026-11-18 (macOS).

#!/usr/bin/env bash
# Usage: ./ai-visibility-check.sh example.com
d="${1:?domain}"
UA="Mozilla/5.0 (compatible; my-check/1.0)"
echo "== robots.txt"
curl -sL -A "$UA" -o /tmp/robots.$$ -w "HTTP %{http_code}  type %{content_type}  final %{url_effective}\n" "https://$d/robots.txt"
echo "-- groups that name an AI crawler:"
grep -iE -A3 '^user-agent:[[:space:]]*(GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|Google-Extended|CCBot)' /tmp/robots.$$ || echo "(none named; the '*' group applies)"
echo "-- wildcard group:"; grep -iE -A4 '^user-agent:[[:space:]]*\*' /tmp/robots.$$ | head -8
echo "-- sitemap lines:"; grep -iE '^sitemap:' /tmp/robots.$$ || echo "(none)"
echo "== llms.txt"
curl -sL -A "$UA" -o /tmp/llms.$$ -w "HTTP %{http_code}  type %{content_type}  final %{url_effective}  bytes %{size_download}\n" "https://$d/llms.txt"
head -c 300 /tmp/llms.$$; echo
echo "== sitemap.xml"
curl -sL -A "$UA" -o /tmp/sm.$$ -w "HTTP %{http_code}  type %{content_type}\n" "https://$d/sitemap.xml"
grep -ocE '<(urlset|sitemapindex)' /tmp/sm.$$ | sed 's/^/xml root tags found: /'
rm -f /tmp/robots.$$ /tmp/llms.$$ /tmp/sm.$$

Reading the output: a named group with Disallow: / blocks that bot, otherwise the * group decides. For llms.txt, a text/html type or a final URL that is not /llms.txt means no file; real ones start with # or a Markdown link. For the sitemap, xml root tags found: 1 or more means XML; 0 with HTTP 200 means a page, or a sitemap declared elsewhere.

It does not weigh competing rules (longest match wins, Allow beats Disallow on a tie, per RFC 9309) and cannot see a firewall. For precedence use a tester such as the free Robots.txt Tester. The free SEO audit runs these checks and the llms.txt Generator drafts a file; in the first study our audit could not read 27 of 100 homepages behind bot protection, so confirm anything it flags.

Methodology and data

  • •Sample: the fixed list of 100 domains from our first study.
  • •Robots.txt and llms.txt: the responses saved on 2026-11-17 (UTC). Blocking means a Disallow wins for the path in the group that applies to the bot (its own group, else *; longest match; Allow wins ties), for the paths /, /llms.txt and /pricing. 96 of 100 robots.txt files were readable. Sub-path blocks other than these three were not examined.
  • •llms.txt classification: the classifyTextResource function from SEOBolt at commit 962ffba (after PR #172), applied to the saved responses; no new request to those URLs was needed. The four content checks use the same regular expressions as the audit's llms-txt-ok rule.
  • •Sitemaps: fetched live on 2026-11-18, only the first declared URL per site, with the same user agent as before and one retry with a desktop Chrome user agent when the answer was not 200, 404 or 410. "Valid" means the body contains <urlset or <sitemapindex; a gzip sitemap was not decoded.
  • •Not tested: citation or traffic effects, WAF or CDN behaviour, whether models read llms.txt, files that sitemap indexes point to.
  • •Correction to our first post: its maximum llms.txt size ("about 1 MB") should read 2.5 MB (twilio.com).
  • •Data and script: cut2.csv and cut2.mjs. The script reads the saved responses of the first study; the scripts to collect them are in its data folder.
  • •Disclosure: we build SEOBolt.

Frequently asked questions

Should I block GPTBot and ClaudeBot?

That is a choice about training data, separate from letting the search and user-fetch bots through.

If I allow OAI-SearchBot, will ChatGPT cite me?

OpenAI says an opted-out site will not be shown in ChatGPT search answers; allowing guarantees nothing, and we did not measure citations.

Do I need an llms.txt?

The vendors we read do not document one as a requirement. If you add one, check that the URL returns a text file, not your homepage.

My robots.txt allows everything. Why would a bot still not see my site?

A WAF, CDN rule or login wall can stop a crawler robots.txt allows. We did not test any such setup.

Is the 100-site sample representative?

No: 100 hand-picked well-known domains, skewed toward tech, on one day.

Free tools for this: AI Visibility Checker, Robots.txt Tester, llms.txt Generator. No sign-up needed.

Check these issues on your site

Run a free homepage audit for technical SEO and AI-search blockers. No signup required.