How do I check whether my site is visible in AI search (GEO)?
AI search visibility is two separate questions. First, can the AI crawlers reach and read your pages: robots.txt, firewall, status codes and text in the HTML. Second, do AI answers cite you for the questions your buyers ask. Check the first with a crawler test and the second by sampling prompts. No tool can promise citations.
By SEOBolt. Published , updated .
Why this happens
People use "GEO" (generative engine optimization) or "AEO" for the work of appearing in answers from ChatGPT, Claude, Perplexity and Google's AI features. The work splits cleanly in two, and most confusion comes from mixing the halves up. Access problems are binary and fixable in an afternoon: a bot is blocked or it is not. Citation outcomes are probabilistic and depend on what else exists on the web for the same question.
The access half matters because each AI product runs several different crawlers for different purposes. OpenAI documents OAI-SearchBot (search), GPTBot (training) and ChatGPT-User (actions a user triggers). Anthropic documents Claude-SearchBot, Claude-User and ClaudeBot. Perplexity documents PerplexityBot and Perplexity-User. Blocking the wrong one, or blocking all of them with a firewall rule, is the most common reason a site is invisible.
Google's own position is that its AI features follow normal search eligibility: a page must be indexed and eligible to show with a snippet, and Google says there are no additional technical requirements and no special files or markup needed for AI Overviews or AI Mode.
How to check it, step by step
None of these steps need a paid tool.
1. Read your robots.txt for each AI crawler
Open https://yourdomain/robots.txt. For each token (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User) find the group that applies to it. If none names the bot, the User-agent: * group applies. A Disallow: / in that group blocks them all.
2. Fetch a page the way each bot would
robots.txt only states intent. A firewall or CDN bot rule can still return 403 to a crawler. Run curl -sI -A "<published user-agent string>" https://yourdomain/ using the string from the vendor's documentation, and expect HTTP 200. OpenAI and Perplexity publish IP ranges you can allow-list; Anthropic lists its ranges at claude.com/crawling/bots.json.
3. Make sure the text is in the HTML
Run curl -s https://yourdomain/page and search the output for your main sentence. Crawlers that do not run JavaScript only see what the server returns. If the content appears only after scripts run, render it on the server.
4. Check the page is indexable
Google's AI features can only link pages that are indexed and eligible to appear with a snippet. Look for a noindex tag or header, a canonical pointing elsewhere, and a nosnippet directive. In Search Console, URL Inspection shows the indexed state.
5. Lead each page with the answer
Put the direct answer to the page's question in the first paragraph, in plain text, followed by the reasoning. It helps people first; it also gives a model something clean to quote. Keep structured data identical to the visible text.
6. Sample prompts and log the sources
Write 5 to 10 buyer prompts, ask them with search on over several days, and record cited domains. The ChatGPT citation page linked below shows the method in detail.
The AI crawlers to check and what blocking each one does
| Crawler | Vendor | Purpose | What the vendor says about blocking it |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Search | Sites opted out will not be shown in ChatGPT search answers, though can still appear as navigational links |
| ChatGPT-User | OpenAI | Fetches pages for a user's request | OpenAI says robots.txt rules may not apply to user-initiated actions |
| GPTBot | OpenAI | Training | Disallowing it indicates content should not be used to train OpenAI's foundation models |
| Claude-SearchBot | Anthropic | Search quality | May reduce visibility and accuracy in user search results |
| Claude-User | Anthropic | Fetches pages for a user's question | May reduce visibility for user-directed web search |
| ClaudeBot | Anthropic | Training | Signals future material should be excluded from training datasets |
| PerplexityBot | Perplexity | Search results | Perplexity recommends allowing it to appear in its search results |
| Perplexity-User | Perplexity | Fetches pages for a user's question | Perplexity says this fetcher generally ignores robots.txt rules |
When to use SEOBolt for this
SEOBolt's free AI Crawler Access Checker at /tools/ai-crawler-access-checker runs steps 1 and 2 for you: a robots.txt verdict per crawler plus a live fetch with each crawler's published user-agent string, which exposes firewall blocks that robots.txt does not show. The free ChatGPT Visibility Checker and AI Visibility Checker add noindex, canonical and schema checks on the homepage.
In a crawl, SEOBolt reports AI-crawler blocks once per site. The free audit at /free-audit shows a separate LLM Search score next to the Google SEO score. For the citation half, the seo_ai_citations tool samples buyer prompts through a search-enabled model and reports which domains were cited.
MCP tools involved: seo_audit, seo_issues, seo_ai_citations.
When SEOBolt is not the right tool
SEOBolt's free checks measure whether crawlers can reach and read a site; they do not query ChatGPT, Claude or Perplexity and promise no citations. For continuous tracking across several AI engines, use a monitoring product built for that.
Prompt to give your AI assistant
With the SEOBolt MCP server connected (setup), copy this into Claude, ChatGPT, Cursor or Gemini CLI:
Assign to SEOBolt: use seo_audit and seo_issues on my project and list every issue that stops OAI-SearchBot, ChatGPT-User, Claude-SearchBot or PerplexityBot from reading my pages, with the fix for each. Do not change anything. Show me what you did so I can approve the fixes.
Frequently asked questions
What is GEO and how is it different from SEO?
GEO means appearing in answers generated by AI products such as ChatGPT, Claude, Perplexity and Google's AI features. It builds on SEO: Google says the same fundamentals apply and that its AI features have no extra technical requirements.
Do I need llms.txt to be visible in AI search?
No evidence says you do. The crawler documentation from OpenAI and Anthropic does not list it, and Google says no special AI files are needed. See the llms.txt answer for details.
If my robots.txt allows the bots, am I done?
No. A firewall, CDN rule or bot-protection setting can still block them. Test with a live fetch using the bot's user-agent string, not only robots.txt.
How long do robots.txt changes take to count?
OpenAI and Perplexity both say their systems can take up to about 24 hours to reflect robots.txt changes.
Sources
- OpenAI: Overview of OpenAI crawlers, read 13 November 2026
- Anthropic Help Center: Does Anthropic crawl data from the web, and how can site owners block the crawler?, read 13 November 2026
- Perplexity: Perplexity crawlers, read 13 November 2026
- Google Search Central: AI features and your website, read 13 November 2026