Does stanford.edu block ChatGPT, Claude and Perplexity?

stanford.edu allows all 12 AI crawlers we test.

Read from stanford.edu's own robots.txt on . Category: Education and reference. Profile: Open to every AI crawler we test.

What it means

Nothing in stanford.edu's robots.txt stops the OpenAI, Anthropic, Perplexity, Google, Common Crawl or Apple crawlers from reading the homepage. Whether an assistant cites the site then depends on its content and authority, not on crawler access.

robots.txt is a request, not a lock: it does not show what a crawler does in practice, and it cannot reveal firewall rules. Blocking is often a deliberate business decision.

Crawler by crawler

robots.txt verdict for each AI crawler on stanford.edu, for the path /
CrawlerVerdict and the rule behind it
OAI-SearchBotOpenAI · searchAllowedthe wildcard "User-agent: *" group: Allow: /
ChatGPT-UserOpenAI · user requestAllowedthe wildcard "User-agent: *" group: Allow: /
GPTBotOpenAI · model trainingAllowedthe wildcard "User-agent: *" group: Allow: /
Claude-SearchBotAnthropic · searchAllowedthe wildcard "User-agent: *" group: Allow: /
Claude-UserAnthropic · user requestAllowedthe wildcard "User-agent: *" group: Allow: /
ClaudeBotAnthropic · model trainingAllowedthe wildcard "User-agent: *" group: Allow: /
PerplexityBotPerplexity · searchAllowedthe wildcard "User-agent: *" group: Allow: /
Perplexity-UserPerplexity · user requestAllowedthe wildcard "User-agent: *" group: Allow: /
Google-ExtendedGoogle · model trainingAllowedthe wildcard "User-agent: *" group: Allow: /
CCBotCommon Crawl · model trainingAllowedthe wildcard "User-agent: *" group: Allow: /
ApplebotApple · searchAllowedthe wildcard "User-agent: *" group: Allow: /
Applebot-ExtendedApple · model trainingAllowedthe wildcard "User-agent: *" group: Allow: /

What each crawler does, and where its token is documented, is on the AI Crawler Access Checker.

Other signals AI tools read

llms.txt
Not found: /llms.txt returns 404.
Sitemap
robots.txt lists 1 sitemap. /sitemap.xml is a valid sitemap.
Structured data on the homepage
No JSON-LD found.
Open Graph and canonical
og:title present; canonical link present.

Compared with education and reference

Of the 22 education and reference sites we read, 2 (9%) block GPTBot, 2 (9%) block ClaudeBot and 2 (9%) block PerplexityBot. stanford.edu blocks 0 of 12.

If you run a site like this

Two separate choices: whether AI assistants may find and cite your pages (search crawlers) and whether your content may be used for training. A robots.txt can answer them differently.

Stay citable in AI search

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

Opt out of training only

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Questions

Does stanford.edu block GPTBot?

No. the wildcard "User-agent: *" group: Allow: / Read 19 November 2026.

Does stanford.edu block ClaudeBot?

No. the wildcard "User-agent: *" group: Allow: / Read 19 November 2026.

Does stanford.edu have an llms.txt file?

Not found: /llms.txt returns 404. Read 19 November 2026.

Check your own site the same way

Free audit, no signup: homepage, robots.txt, sitemap.xml and llms.txt, with the fix for each issue.

No signup and no payment details needed. Checks the homepage, robots.txt, sitemap.xml and llms.txt.

Snapshot of 19 November 2026. Data for all sites: CSV. Something wrong, or want this page removed? Email support@seobolt.app. SEOBolt builds the audit tool this data comes from.