Skip to content

AI Crawler Checker

AI crawler checker tests supplied robots.txt rules against a path for the site’s AI crawler token list. It merges matching product-token groups and applies the most specific matching path rule.

Found a problem, or something that could be better? We read every message and we fix things quickly.

AI crawler checker tests supplied robots.txt rules against a path for the site’s AI crawler token list. It merges matching product-token groups and applies the most specific matching path rule.

The page reports allowed or blocked by these rules with the matching line. It does not fetch a website, verify a live bot, inspect server headers or guarantee that an operator honors the result.

How to use

  1. Paste the robots.txt text you want to inspect.
  2. Enter a root-relative path with an optional query string.
  3. Run and review each crawler’s matched rule and ignored-line diagnostics.

Frequently asked questions

Does it check my live website?

No. It analyzes only pasted text. A live website can serve different robots.txt content, redirects, HTTP status codes or headers.

Which rule wins?

The longest matching path wins. An Allow wins an equal-specificity tie; exact product-token groups merge, with the wildcard group used only when no specific group matches.

Is blocked the same as protected?

No. robots.txt is voluntary crawler guidance, not authentication or access control. It does not remove already collected data.

Does Google-Extended block Google Search?

It is a separate control token rather than Googlebot itself. Vendor-specific usage policies and user-triggered fetches need the operator’s current documentation, beyond this text matcher.

Path comparison

Matching is case-sensitive for paths and case-insensitive for crawler tokens. Wildcard * and terminal $ are supported; encoded reserved bytes remain distinct from literal separators.

For example, /private/public can override a shorter /private block. /a%2Fb does not become /a/b; unreserved encoded ASCII is normalized before comparison.

What the report omits

Unsupported records are reported by line rather than interpreted as invented blocking rules. Up to 512,000 input characters and 5,000 path rules are accepted.

The crawler list is shared with the existing robots generator. This tool evaluates its supplied tokens, not an exhaustive census of crawlers or a legal opt-out verification.