Does washingtonpost.com block ChatGPT, Claude and Perplexity?
washingtonpost.com blocks 4 of 12 AI crawlers we test, including ClaudeBot, PerplexityBot, CCBot.
Read from washingtonpost.com's own robots.txt on . Category: News and media. Profile: Blocks at least one AI search crawler.
What it means
washingtonpost.com disallows PerplexityBot, an AI search crawler. The matching assistant has less of the site to index and cite. It also blocks other crawlers listed below.
robots.txt is a request, not a lock: it does not show what a crawler does in practice, and it cannot reveal firewall rules. Blocking is often a deliberate business decision.
Crawler by crawler
| Crawler | Verdict and the rule behind it |
|---|---|
OAI-SearchBotOpenAI · search | Allowedthe wildcard "User-agent: *" group: no rule matches /, so it is allowed. |
ChatGPT-UserOpenAI · user request | Allowedthe wildcard "User-agent: *" group: no rule matches /, so it is allowed. |
GPTBotOpenAI · model training | Allowedthe wildcard "User-agent: *" group: no rule matches /, so it is allowed. |
Claude-SearchBotAnthropic · search | Allowedthe wildcard "User-agent: *" group: no rule matches /, so it is allowed. |
Claude-UserAnthropic · user request | Allowedthe wildcard "User-agent: *" group: no rule matches /, so it is allowed. |
ClaudeBotAnthropic · model training | Blockedits own "ClaudeBot" group: Disallow: / |
PerplexityBotPerplexity · search | Blockedits own "PerplexityBot" group: Disallow: / |
Perplexity-UserPerplexity · user request | Allowedthe wildcard "User-agent: *" group: no rule matches /, so it is allowed. |
Google-ExtendedGoogle · model training | Allowedthe wildcard "User-agent: *" group: no rule matches /, so it is allowed. |
CCBotCommon Crawl · model training | Blockedits own "CCBot" group: Disallow: / |
ApplebotApple · search | Allowedthe wildcard "User-agent: *" group: no rule matches /, so it is allowed. |
Applebot-ExtendedApple · model training | Blockedits own "Applebot-Extended" group: Disallow: / |
What each crawler does, and where its token is documented, is on the AI Crawler Access Checker.
Other signals AI tools read
- llms.txt
- Could not be read (HTTP no response).
- Sitemap
- robots.txt lists 7 sitemaps. /sitemap.xml could not be read.
- Structured data on the homepage
- The homepage could not be read (HTTP no response).
- Open Graph and canonical
- Homepage not readable.
Compared with news and media
Of the 14 news and media sites we read, 8 (57%) block GPTBot, 11 (79%) block ClaudeBot and 10 (71%) block PerplexityBot. washingtonpost.com blocks 4 of 12.
If you run a site like this
Two separate choices: whether AI assistants may find and cite your pages (search crawlers) and whether your content may be used for training. A robots.txt can answer them differently.
Stay citable in AI search
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /Opt out of training only
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /Questions
Does washingtonpost.com block GPTBot?
No. the wildcard "User-agent: *" group: no rule matches /, so it is allowed. Read 19 November 2026.
Does washingtonpost.com block ClaudeBot?
Yes. its own "ClaudeBot" group: Disallow: / Read 19 November 2026.
Does washingtonpost.com have an llms.txt file?
Could not be read (HTTP no response). Read 19 November 2026.
Check your own site the same way
Free audit, no signup: homepage, robots.txt, sitemap.xml and llms.txt, with the fix for each issue.
No signup and no payment details needed. Checks the homepage, robots.txt, sitemap.xml and llms.txt.
Snapshot of 19 November 2026. Data for all sites: CSV. Something wrong, or want this page removed? Email support@seobolt.app. SEOBolt builds the audit tool this data comes from.