SitemapForge

Robots.txt checker

Test any list of URLs against a robots.txt and see the exact line that lets each one through or turns it away.

Loading…

How a crawler decides

  1. Pick a group. The crawler looks for the group whose User-agent line matches its token most specifically. Googlebot-Image uses a Googlebot-Image group if there is one, otherwise Googlebot, otherwise *. It follows only that group.
  2. Find every matching rule. Each Allow and Disallow path is compared with the start of the URL path (plus query string). * matches anything; a trailing $ means "ends here".
  3. Longest match wins. Of the matching rules, the one with the longest path decides. Allow: /search/help (12 characters) beats Disallow: /search (7). On a tie, Allow wins.
  4. No match means allowed.

The highlighted source view in the tool shows this happening: tinted lines are the chosen group, orange lines are the rules that actually decided a URL.

Questions people ask

How do I check if a URL is blocked by robots.txt?
Load the site's robots.txt (or paste it), pick a crawler such as Googlebot, and list the URLs to test. Each one is marked Allowed or Blocked with the exact line that decided it.
Does this match Google's own tester?
It follows the same public rules: RFC 9309 and Google's robots.txt specification. The most specific user-agent group applies, the longest matching rule wins, Allow wins ties, and * and $ work as wildcards. Search Console's robots.txt report is still the final word for Googlebot.
What happens if robots.txt returns 404 or 500?
A 4xx response (including 404) means there are no restrictions. A 5xx response or a timeout makes Google treat the whole site as disallowed for a while, which is why a broken robots.txt can stop crawling of a healthy site.
Why is my page blocked when I only disallowed a folder?
Rules match from the start of the path as prefixes. Disallow: /blog also blocks /blog-news and /blogger. Add a trailing slash (/blog/) to limit it to the folder.