Robots Txt Generator
Build crawler rules and an optional sitemap declaration for your website.
Your workspace
Processed in your browserThis tool generates a standard robots.txt file, which acts as a set of instructions for web crawlers. It helps you specify which parts of your site should be crawled and where your sitemap is located.
Creating your robots.txt
- Specify the user agent (e.g., * for all crawlers). Use the record editors to add paths you want to disallow or allow.
- Provide the absolute URL of your sitemap. The tool will assemble these into a standard text format that you can save as robots.txt in your site root.
Limitations of robots.txt
Robots.txt is a directive for compliant crawlers; it does not physically secure private files or prevent indexing if a page is linked elsewhere. Always use password protection or noindex tags for sensitive content.
Example configuration
User agent: *, Disallow: /private/, Allow: /private/public/, Sitemap: https://example.com/sitemap.xml
User-agent: * Allow: /private/public/ Disallow: /private/ Sitemap: https://example.com/sitemap.xml
Questions about robots.txt
Does robots.txt prevent a page from being indexed?
No. Disallowing a page in robots.txt prevents crawling, but if other sites link to it, the URL may still appear in search results without a description.
Can I use wildcards in my paths?
Yes, standard robots.txt implementations support wildcards like * to match sequences of characters in path directives.
Continue your workflow
Explore more tools in this category for your next step.
Anchor Text Audit
List anchor text and resolved destinations from pasted HTML, including empty and image-only links.
Bulk HTTP Status Checker
Check up to five public HTTP or HTTPS URLs and inspect the returned status; private network addresses are not accepted.
Canonical Tag Generator
Create one escaped canonical link tag for an absolute page URL. This tool outputs the HTML snippet needed to signal your preferred content version to search engines.
Canonical URL Checker
Inspect the canonical tag and compare its target with the page URL you provide. This tool checks for tag presence, absolute resolution, and self-reference.