Robots.txt generator
The first three replace your groups. The AI button adds one group with commonly published AI crawler names and keeps the rest; the list changes over time, so check it now and then.
Import an existing robots.txt
* means every crawler. Separate several names with commas to give them the same rules.
Paths start with / and are case-sensitive. * matches any characters and $ marks the end of the URL, so /*.pdf$ blocks every PDF. Crawl-delay (seconds between requests) is ignored by Google; some other search engines respect it.
No problems found.
Upload it as robots.txt to the root of the host, e.g. https://example.com/robots.txt. Each subdomain needs its own file.
Test a URL
Checks the robots.txt above the way Google reads it. A group naming the crawler exactly beats User-agent: *, groups with the same name are merged, and a crawler with no group of its own falls back to *, not to a related name (Googlebot-Image does not use a Googlebot group). Within the group the longest matching rule decides and Allow wins a tie. A full URL is reduced to its path and query.
robots.txt is a public request, not a lock. Anyone can read it and badly behaved bots ignore it, so never list secret paths there. It also does not remove pages from search results: a blocked URL can still be indexed from links. To keep a page out of search, use a noindex tag and leave the page crawlable; to protect it, use a password.
How robots.txt works
Before crawling a site, well-behaved bots fetch /robots.txt and look for the group that matches their name, falling back to User-agent: *. Inside the group, Disallow lists paths they should skip and Allow carves exceptions out of them. The file controls crawling, not indexing, and only bots that choose to follow it do.
How to use it
- 1 Pick a starting template or edit the example: one group per crawler (or * for all), with Disallow and Allow lines.
- 2 Add your sitemap address and read the checks: a path without a leading slash or a sitemap without https:// is flagged right away.
- 3 Copy the text or download robots.txt and upload it to the root of your site, then open yoursite/robots.txt to confirm it loads.
Which rule wins
Major search engines do not read rules top to bottom. For each URL they take the most specific match, meaning the longest path; when an Allow and a Disallow are equally long, Allow wins. So Disallow: /admin/ with Allow: /admin/help/ keeps the help pages crawlable whatever order you write them in. A crawler also follows only one group: Googlebot obeys a Googlebot group and ignores the * group entirely.
Blocking is not the same as noindex
A URL blocked in robots.txt can still appear in search results, usually as a bare link with no description, because other pages link to it. Worse, if you add a noindex tag to a page and also block it, the crawler never fetches the page and never sees the tag. To remove a page from search, allow crawling and use noindex; block it only once it has dropped out.
Mistakes that block more than you meant
Rules are prefixes: Disallow: /blog also blocks /blog-tips/ and /blogger.html; write /blog/ if you mean the folder. Disallow: / on its own blocks the whole site, a classic leftover from a staging server. Blocking CSS and JavaScript folders stops search engines from rendering your pages properly. And paths are case-sensitive: /Admin/ does not cover /admin/. To see how a particular URL is treated, enter it in the test box under the output: it names the group and the rule that decide.
AI crawlers and control tokens
Several AI companies publish the user-agent names they use to collect training data and document robots.txt as the way to opt out. Some names are not separate crawlers but control tokens: Google-Extended and Applebot-Extended never fetch pages; they only tell the company whether content its search crawler has already collected may be used to train its AI models (for Google, also to ground Gemini answers). Blocking them does not affect your search ranking, but it does not keep your pages out of AI features inside search either: Google’s AI Overviews and AI Mode run on Googlebot and are controlled with nosnippet or noindex, not with Google-Extended. Names change and new ones appear, so treat the preset as a starting point and check each provider’s current documentation.