Robots Txt Generator

Build crawler rules and an optional sitemap declaration for your website.

Your workspace

Processed in your browser

Disallow paths

Entry 1
Entry 2

2 entries

Import or copy multiple entries

Allow paths

Entry 1

1 entries

Import or copy multiple entries

Review the example or enter your values.

This tool generates a standard robots.txt file, which acts as a set of instructions for web crawlers. It helps you specify which parts of your site should be crawled and where your sitemap is located.

Creating your robots.txt

  1. Specify the user agent (e.g., * for all crawlers). Use the record editors to add paths you want to disallow or allow.
  2. Provide the absolute URL of your sitemap. The tool will assemble these into a standard text format that you can save as robots.txt in your site root.

Limitations of robots.txt

Robots.txt is a directive for compliant crawlers; it does not physically secure private files or prevent indexing if a page is linked elsewhere. Always use password protection or noindex tags for sensitive content.

Example configuration

User agent: *, Disallow: /private/, Allow: /private/public/, Sitemap: https://example.com/sitemap.xml

User-agent: * Allow: /private/public/ Disallow: /private/ Sitemap: https://example.com/sitemap.xml

Questions about robots.txt

Does robots.txt prevent a page from being indexed?

No. Disallowing a page in robots.txt prevents crawling, but if other sites link to it, the URL may still appear in search results without a description.

Can I use wildcards in my paths?

Yes, standard robots.txt implementations support wildcards like * to match sequences of characters in path directives.

Continue your workflow

Explore more tools in this category for your next step.