Build a valid robots.txt file with presets and custom crawl rules. Copy or download in one click.
Free · No sign-up · Instant results
User-agent: * Disallow:
Every block uses correct User-agent, Allow/Disallow, Crawl-delay, and Sitemap directives — no typos to break crawling.
Start from Allow all, Block all, or WordPress, then add as many User-agent blocks and path rules as you need.
Grab the file with one click and drop it at your domain root. No account, no email, no waiting.
Choose Allow all, Block all, or WordPress to start with sensible defaults — or pick Custom to build from scratch.
Set the bot name (* for all), then add Allow/Disallow paths and an optional crawl-delay for each block.
Paste the full URL to your sitemap.xml so crawlers can discover all your pages.
Click Copy or Download, then upload the file to your domain root at /robots.txt.
robots.txt is a plain-text file at the root of your domain that tells search engine crawlers which URLs they may or may not request. It controls crawling (not indexing). Use it to keep bots out of admin areas, search result pages, and duplicate content, and to point them to your sitemap.
Disallow tells a crawler not to request a path (e.g. Disallow: /admin/). Allow explicitly permits a path, which is useful to carve out an exception inside a disallowed folder (e.g. Disallow: /wp-admin/ plus Allow: /wp-admin/admin-ajax.php). The most specific matching rule wins for Google.
It must live at the root of your domain, reachable at https://yourdomain.com/robots.txt. It cannot be placed in a subfolder. Each subdomain (e.g. blog.yourdomain.com) needs its own robots.txt.
The biggest mistake is accidentally blocking your whole site with 'Disallow: /' on production. Other common errors: using robots.txt to hide pages from search (use a noindex meta tag instead — blocked pages can still be indexed without a snippet), case-sensitivity errors in paths, and forgetting the Sitemap line.
No. robots.txt only blocks crawling. A disallowed URL can still appear in search results if other pages link to it. To keep a page out of the index, allow crawling and add a <meta name="robots" content="noindex"> tag, or return an X-Robots-Tag: noindex header.
SEOBolt's Site Audit automatically checks your robots.txt and crawl config, flags accidentally blocked pages, and verifies your sitemap is reachable. Free for 2 projects.
Run Full Site Audit Free →