Fetch a domain robots.txt and see which bots and paths are allowed or blocked.
Free · No sign-up · Includes OAI-SearchBot, GPTBot, and ClaudeBot
It fetches /robots.txt, groups rules by User-agent, applies the most specific matching rule, and treats Allow as the winner when rule lengths tie.
Yes. OAI-SearchBot, ChatGPT-User, GPTBot, ClaudeBot, and PerplexityBot are included by default, and you can add any other bot name before running the test.
No. robots.txt controls crawling. A blocked URL can still be indexed if other pages link to it. Use noindex for pages that should not appear in search.
Most crawlers treat a missing robots.txt as permission to crawl everything that is otherwise accessible.
SEOBolt crawls your whole site, monitors rankings, and flags technical SEO issues before they cost you traffic. Free for 2 projects.
Run Full SEO Audit Free