Fetch a domain robots.txt and see which bots and paths are allowed or blocked.
Free · No sign-up · Includes GPTBot and ClaudeBot
It fetches /robots.txt, groups rules by User-agent, applies the most specific matching rule, and treats Allow as the winner when rule lengths tie.
Yes. GPTBot, ClaudeBot, and PerplexityBot are included by default, and you can add any other bot name before running the test.
No. robots.txt controls crawling. A blocked URL can still be indexed if other pages link to it. Use noindex for pages that should not appear in search.
Most crawlers treat a missing robots.txt as permission to crawl everything that is otherwise accessible.
SEOBolt crawls your whole site, monitors rankings, and flags technical SEO issues before they cost you traffic. Free for 2 projects.
Run Full SEO Audit Free