robots.txt / sitemap.xml Generator help

robots.txt / sitemap.xml Generator writes the three small files that sit at the root of a site: a robots.txt with allow and disallow rules, a sitemap.xml built from a list of paths with optional lastmod and priority, and a humans.txt credits file.

Steps

  1. Pick the file you want to build with the tabs.
  2. Fill in the fields. For the sitemap, one path per line, with optional comma-separated lastmod, changefreq, and priority.
  3. Copy the output or download it with the right filename.

Common questions

What does the AI crawler block do?

It adds a second group that disallows a set of well-known AI training and assistant crawlers. Note that robots.txt is advisory and not all crawlers honour it.

How should the sitemap paths look?

Start each with a slash. You can add up to three more comma-separated values: a lastmod date, a changefreq word, and a priority between 0 and 1.

Where do these files go?

At the root of your domain, so they are reachable at /robots.txt, /sitemap.xml, and /humans.txt.

Is anything uploaded?

No. The files are built in your browser.

Everything runs in your browser. Nothing you enter or drop is uploaded.

Back to robots.txt / sitemap.xml Generator