Robots Txt Tester

Verify if a specific crawler is permitted to access a URL path based on your pasted robots.txt rules. This tool evaluates standard exclusion protocols.

Your workspace

Processed in your browser

Review the example or enter your values.

How to use Robots Txt Tester

  1. Paste your complete robots.txt content into the robots.txt field.
  2. Enter the specific user-agent string, such as Googlebot, into the Crawler product token field.
  3. Input the specific path or absolute URL you wish to verify in the Path or absolute URL to test field.

When this tool is useful

The robots.txt file acts as the primary instruction manual for web crawlers visiting your site. When you update your directives, it is critical to verify that your exclusions are functioning exactly as intended. This tester allows you to simulate how a search engine bot interprets your rules by processing your raw robots.txt content against a specific user-agent and target URL path. By checking your configuration locally before deployment, you can prevent accidental blocking of important content or unintended exposure of private directories. This utility strictly evaluates the provided text block based on standard path matching and precedence rules used by major search engines.

Practical example

robots.txt: Disallow: /private/ Crawler: Googlebot Path: /private/report/

Allowed: false. The path /private/report/ is blocked because it matches the Disallow rule for /private/.

Understanding Crawler Precedence

When evaluating your robots.txt file, the system follows a strict hierarchy of rules. The most specific path match always takes precedence over less specific ones. For example, a rule targeting /private/files/ will override a general rule for /private/. If multiple rules have the same specificity, the Allow directive wins over the Disallow directive. This tool processes your input using these exact standard logic gates. It is important to remember that this tester only examines the text you provide. It does not check for meta tags, HTTP headers, or server-side authentication, which can also influence how a crawler interacts with your website content.

Managing Crawler Tokens

The Crawler product token field is case-insensitive but must match the identifier used by the bot. If you specify a general wildcard asterisk, the tool applies those rules to any crawler that does not have a more specific block defined. When testing, ensure that you are using the exact token expected by the search engine, such as Bingbot or Googlebot. If you leave this field blank or use an incorrect token, the simulation may not accurately reflect how a specific bot perceives your site. Always verify that your robots.txt file includes the necessary user-agent lines to ensure your intended directives are actually being applied to the correct crawlers.

Method and limits

Matches crawler groups, wildcard paths and end markers; the longest applicable path wins and Allow wins a tie. Only robots.txt rules are evaluated.

Common questions

Does this tool check my live website?

No, this tool does not crawl your live website. It only evaluates the text you manually paste into the input field. This ensures your private configuration remains secure and is never transmitted or scanned by external servers during the testing process.

Why is my path still blocked?

If your path is blocked, check for a Disallow rule that matches the beginning of your path. Remember that robots.txt rules are prefix-based. If you have a rule like Disallow: /private/, any URL starting with that string will be restricted, regardless of what follows.

Does this handle meta robots tags?

No, this tool is strictly for robots.txt files. It does not analyze meta robots tags, X-Robots-Tag HTTP headers, or any other directives found within the HTML of your pages. Those methods are handled independently by search engine crawlers.

Continue your workflow

Explore more tools in this category for your next step.