SitemapForge

Robots.txt generator

Point crawlers at your sitemap and keep them out of cart, search and admin pages, with rules written the way RFC 9309 reads them.

Loading…

How robots.txt and your sitemap work together

robots.txt says where crawlers may go; the sitemap says where you'd like them to go. The Sitemap: line connects the two: every crawler that reads robots.txt, including ones you have never submitted to, learns where your sitemap is.

The two must not contradict each other. A URL that is in your sitemap but disallowed in robots.txt shows up in Search Console as "Submitted URL blocked by robots.txt". Before you publish, run the URLs from your sitemap through the checker.

Rules that trip people up

  • Disallow: with an empty value allows everything. Disallow: / blocks everything.
  • Paths are case-sensitive and match from the start: /Private does not block /private.
  • * matches any run of characters and $ anchors the end, so Disallow: /*.pdf$ blocks PDFs only.
  • A crawler obeys only the single most specific group that names it. If you add a Googlebot group, Googlebot ignores the * group entirely, so repeat any shared rules.
  • Google ignores Crawl-delay, Noindex and Host.

Questions people ask

How do I add my sitemap to robots.txt?
Add a line Sitemap: https://www.example.com/sitemap.xml anywhere in the file, using the full absolute URL. It is independent of user-agent groups, and you can add several lines for several sitemaps or a sitemap index.
Does Disallow remove a page from Google?
No. Disallow stops crawling, not indexing. A blocked URL can still appear in results if other sites link to it, just without a description. To keep a page out of the index, allow crawling and use a noindex meta tag or X-Robots-Tag header, as Google's robots.txt introduction explains.
Where does robots.txt go?
At the root of each host: https://www.example.com/robots.txt. It only applies to that exact protocol, host and port, so blog.example.com needs its own file.
Which rule wins when Allow and Disallow overlap?
Under RFC 9309 the most specific (longest) matching path wins. If an Allow and a Disallow are equally long, Allow wins. Test combinations with the robots.txt checker.
Can I block AI crawlers with robots.txt?
You can ask them. OpenAI (GPTBot), Anthropic (ClaudeBot), Google (Google-Extended), Common Crawl (CCBot) and others publish user-agent tokens and say they honour robots.txt. Blocking Google-Extended does not affect Google Search.