Will blocking GPTBot remove my site from ChatGPT search?

No. OpenAI documents GPTBot and OAI-SearchBot as independent robots.txt settings. Disallowing GPTBot says your content should not be used to train its models; it does not remove you from ChatGPT search. Only sites opted out of OAI-SearchBot are left out of ChatGPT search answers. Allow OAI-SearchBot if you want to be shown.

By SEOBolt. Published , updated .

Why this happens

Many site owners added a blanket block for AI crawlers in 2024 and 2025 to stay out of training data, and some of those blocks also removed them from AI search. The cause is that each vendor runs separate crawlers for separate jobs, and a rule written for one bot name does not touch the others, while a rule written for User-agent: * touches all of them.

OpenAI's crawler documentation states the split directly: a webmaster can allow OAI-SearchBot to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training its generative AI foundation models. Anthropic draws the same line between ClaudeBot (training) and Claude-SearchBot and Claude-User (search and user requests).

There is one more wrinkle. ChatGPT-User and Perplexity-User fetch a page because a person asked about it. OpenAI says robots.txt rules may not apply to those user-initiated actions, and Perplexity says its user fetcher generally ignores robots.txt. A robots.txt rule is therefore a request that well-behaved crawlers honor, not a lock.

How to set it up, step by step

Decide your policy first, then write the rules, then test them.

  1. 1. Decide what you want

    Three common policies: appear in AI search and allow training; appear in AI search but opt out of training; stay out of both. The second is the usual choice for publishers and software companies that want visibility without contributing training data.

  2. 2. Write one group per bot

    To appear in search but opt out of OpenAI and Anthropic training, add groups such as User-agent: GPTBot with Disallow: / and User-agent: ClaudeBot with Disallow: /, and leave OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User and PerplexityBot without a Disallow. Only the group that best matches a crawler applies to it.

  3. 3. Check what User-agent: * does

    If you have a User-agent: * group with Disallow: /, it applies to every crawler that has no group of its own. Give search bots their own group with Allow: / if you want them in.

  4. 4. Allow the published IP ranges in your firewall

    OpenAI and Perplexity publish IP address lists for their bots, and Anthropic publishes ranges at claude.com/crawling/bots.json. If your CDN or web firewall challenges unknown bots, allow those ranges for the search crawlers.

  5. 5. Wait for the change to take effect

    OpenAI says it can take about 24 hours after a robots.txt update for its search systems to adjust, and Perplexity gives the same window.

  6. 6. Test the result

    Fetch /robots.txt, confirm the groups, then test a real URL against each bot with a robots.txt tester and a live request using each bot's user-agent string. Re-test after any CDN or firewall change.

Bot-by-bot: what to block if you want search visibility without training

Training and search crawlers from OpenAI and Anthropic (vendor documentation read 13 November 2026)
GoalOpenAI robots.txt tokenAnthropic robots.txt token
Opt out of model trainingGPTBotClaudeBot
Stay in AI search resultsAllow OAI-SearchBotAllow Claude-SearchBot
Allow fetches a user asks forChatGPT-User (robots.txt rules may not apply)Claude-User (disabling may reduce visibility for user-directed search)

When to use SEOBolt for this

The free robots.txt Tester (/tools/robots-txt-tester) checks a path against Googlebot, OAI-SearchBot, ChatGPT-User, GPTBot, ClaudeBot and PerplexityBot, and the AI Crawler Access Checker (/tools/ai-crawler-access-checker) adds a live fetch with each bot's published user agent. Both are anonymous. In a crawl, SEOBolt reports when OAI-SearchBot or ChatGPT-User are blocked from the homepage, /llms.txt or /pricing, and when Claude or Perplexity crawlers are blocked.

SEOBolt reads and reports; it does not edit your robots.txt. Your agent can propose the edit from seo_issues and you approve it in your own repository.

MCP tools involved: seo_issues, seo_fix_suggestions.

When SEOBolt is not the right tool

A robots.txt rule cannot stop a crawler that ignores it, and it does not stop user-initiated fetches that OpenAI and Perplexity say may bypass it. If content must be private, put it behind a login.

Prompt to give your AI assistant

With the SEOBolt MCP server connected (setup), copy this into Claude, ChatGPT, Cursor or Gemini CLI:

Assign to SEOBolt: use seo_issues to find whether my robots.txt blocks OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User or PerplexityBot. Propose a robots.txt that opts out of GPTBot and ClaudeBot training but keeps the search bots allowed. Show me what you did and the proposed file so I can approve it.

Frequently asked questions

Does blocking ClaudeBot remove me from Claude search?

Per Anthropic's documentation, ClaudeBot is its training crawler. Disabling Claude-SearchBot or Claude-User is what may reduce visibility in Claude's search and user-directed search.

If I allow OAI-SearchBot, will ChatGPT cite me?

Allowing it is a precondition, not a promise. OpenAI recommends allowing OAI-SearchBot and its published IP ranges so a site can appear in search results; whether a page is used in an answer depends on the question and on other pages.

Does robots.txt keep a page out of Google?

No. Google says robots.txt manages crawling and is not a mechanism for keeping a page out of Google; use noindex or password protection for that.

What is Google-Extended?

A robots.txt token Google documents for limiting use of content for AI training and grounding in some of its other systems. Google says AI Overviews and AI Mode in Search are controlled through Googlebot rules and snippet controls.

Sources

Related