# How do I test whether my robots.txt blocks Googlebot or AI crawlers? Fetch https://yourdomain/robots.txt, find the group whose user-agent token best matches the crawler, and test the URL path against that group's Allow and Disallow rules. Only one group applies to a given crawler, and User-agent: * applies only when no specific group matches. Then confirm with a live request, because a firewall can block what robots.txt allows. By SEOBolt. Published 13 November 2026, updated 13 November 2026. ## Why this happens A robots.txt file is a plain text list of groups. Each group starts with one or more User-agent lines and is followed by Allow and Disallow rules. Google's specification says that only one group is valid for a particular crawler: the group with the most specific user agent that matches. Groups for other bots, and the wildcard group, are ignored once a more specific group exists. People get surprising results because they assume rules accumulate, when a specific group replaces the wildcard group rather than adding to it. The file also has failure modes that have nothing to do with its rules. It must sit at the top level of the host, so https://example.com/robots.txt does not govern https://blog.example.com. Google treats most 4xx responses as if no robots.txt existed, which means no restrictions. For a 5xx error it stops crawling for 12 hours and then uses the last good copy for up to 30 days. And it ignores anything beyond 500 KiB. Finally, robots.txt controls crawling, not indexing. Google says a disallowed URL can still be indexed without a snippet if other sites link to it; use noindex or authentication to keep a page out of results. ## How to test it, step by step 1. **Fetch the file and check the response.** Run curl -si https://yourdomain/robots.txt. Expect HTTP 200, a text/plain content type and your rules. A redirect chain, a 5xx error or an HTML error page changes how crawlers treat the file. Repeat for every subdomain that serves pages. 2. **List the user-agent tokens you care about.** Googlebot for Google Search; OAI-SearchBot, ChatGPT-User and GPTBot for OpenAI; Claude-SearchBot, Claude-User and ClaudeBot for Anthropic; PerplexityBot and Perplexity-User for Perplexity. Tokens are case-insensitive; paths are case-sensitive. 3. **Find the one group that applies to each token.** Look for a group naming the token exactly. If none, the User-agent: * group applies. If a specific group exists, ignore every other group for that crawler. 4. **Test the path against that group.** Compare the URL path (including the leading slash) with the group's Allow and Disallow lines. A tester does this for you. In Search Console, URL Inspection shows a "Crawl allowed?" verdict for a specific URL, so you can see whether robots.txt blocks it for Google. 5. **Confirm with a live request.** Run curl -sI -A "" https://yourdomain/page. A 403 or a challenge page means a firewall or CDN rule is blocking the bot even though robots.txt allows it. 6. **Re-test after every change.** Cache lifetimes matter. Google generally caches robots.txt for up to 24 hours; OpenAI and Perplexity say their systems can take about a day to reflect changes. Test again the next day. ## What different responses for robots.txt mean for Google How Googlebot treats robots.txt by HTTP status (Google's robots.txt specification, read 13 November 2026) | Response | How Google treats it | | --- | --- | | 2xx | Processes the file as served | | 3xx redirect | Follows at least five hops, then treats it as a 404 | | 4xx except 429 | Treats it as if no robots.txt exists, so no crawl restrictions | | 5xx | Stops crawling for 12 hours, then uses the last good copy for up to 30 days; with no cached copy it assumes no restrictions | | File over 500 KiB | Content after the limit is ignored | ## When to use SEOBolt for this SEOBolt's free robots.txt Tester at /tools/robots-txt-tester checks any path against Googlebot, OAI-SearchBot, ChatGPT-User, GPTBot, ClaudeBot and PerplexityBot and names the rule that matched, with no signup. The AI Crawler Access Checker adds the live user-agent fetch from step 5. In a crawl, SEOBolt flags pages blocked by robots.txt and AI-crawler blocks across the site; the free audit also flags a missing robots.txt and a missing Sitemap line. MCP tools involved: `seo_issues`, `seo_fix_suggestions`. **When SEOBolt is not the right tool:** It reads the live file and reports; it does not change robots.txt, and it cannot see rules a CDN applies outside robots.txt unless the live fetch is blocked too. ## Prompt to give your AI assistant With the SEOBolt MCP server connected (https://seobolt.app/mcp-seo): ```text Assign to SEOBolt: with seo_issues, list every page my latest crawl found blocked by robots.txt or noindex, and say whether each block looks intentional. Then give me the corrected robots.txt as a diff. Show me what you did so I can approve the change. ``` ## FAQ ### Does Disallow in robots.txt remove a page from Google? No. Google says a disallowed URL can still be indexed without a snippet if other pages link to it. Use a noindex rule or password protection to keep a page out of search results. ### Which group wins if both User-agent: * and User-agent: Googlebot exist? Googlebot uses only the Googlebot group. The wildcard group is ignored for it because a more specific group exists. ### Do I need a robots.txt at all? No. Without one, crawlers assume there are no restrictions. You need one to manage crawl traffic, to block specific paths, or to declare a sitemap. ### Why does a bot still fetch my blocked pages? Compliance is voluntary. Reputable crawlers honor robots.txt, others may not, and user-initiated fetchers can bypass it. Use authentication for anything private. ## Sources - [Google Search Central: How Google interprets the robots.txt specification](https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec), read 13 November 2026 - [Google Search Central: Introduction to robots.txt](https://developers.google.com/search/docs/crawling-indexing/robots/intro), read 13 November 2026 - [OpenAI: Overview of OpenAI crawlers](https://platform.openai.com/docs/bots), read 13 November 2026 - [Anthropic Help Center: Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518), read 13 November 2026 - [Perplexity: Perplexity crawlers](https://docs.perplexity.ai/guides/bots), read 13 November 2026 ## Related - [Will blocking GPTBot remove my site from ChatGPT search?](https://seobolt.app/solutions/blocking-gptbot-chatgpt-search) - [How do I check whether Google has indexed my pages?](https://seobolt.app/solutions/check-if-google-indexed-my-pages) - [robots.txt Tester (free tool)](https://seobolt.app/tools/robots-txt-tester) - [robots.txt Generator (free tool)](https://seobolt.app/tools/robots-generator) - [All problems and solutions](https://seobolt.app/solutions) HTML version: https://seobolt.app/solutions/test-robots-txt-googlebot-and-ai-crawlers