Robots.txt Generator - Create Allow, Disallow and Sitemap Rules

Tools / SEO Tools / Robots.txt Generator

Robots.txt generator

Start from

The first three replace your groups. The AI button adds one group with commonly published AI crawler names and keeps the rest; the list changes over time, so check it now and then.

Import an existing robots.txt

Paths start with / and are case-sensitive. * matches any characters and $ marks the end of the URL, so /*.pdf$ blocks every PDF. Crawl-delay (seconds between requests) is ignored by Google; some other search engines respect it.

Checks

No problems found.

robots.txt

            

Upload it as robots.txt to the root of the host, e.g. https://example.com/robots.txt. Each subdomain needs its own file.

Test a URL

    Checks the robots.txt above the way Google reads it. A group naming the crawler exactly beats User-agent: *, groups with the same name are merged, and a crawler with no group of its own falls back to *, not to a related name (Googlebot-Image does not use a Googlebot group). Within the group the longest matching rule decides and Allow wins a tie. A full URL is reduced to its path and query.

    robots.txt is a public request, not a lock. Anyone can read it and badly behaved bots ignore it, so never list secret paths there. It also does not remove pages from search results: a blocked URL can still be indexed from links. To keep a page out of search, use a noindex tag and leave the page crawlable; to protect it, use a password.

    How robots.txt works

    Before crawling a site, well-behaved bots fetch /robots.txt and look for the group that matches their name, falling back to User-agent: *. Inside the group, Disallow lists paths they should skip and Allow carves exceptions out of them. The file controls crawling, not indexing, and only bots that choose to follow it do.

    How to use it

    1. 1 Pick a starting template or edit the example: one group per crawler (or * for all), with Disallow and Allow lines.
    2. 2 Add your sitemap address and read the checks: a path without a leading slash or a sitemap without https:// is flagged right away.
    3. 3 Copy the text or download robots.txt and upload it to the root of your site, then open yoursite/robots.txt to confirm it loads.

    Which rule wins

    Major search engines do not read rules top to bottom. For each URL they take the most specific match, meaning the longest path; when an Allow and a Disallow are equally long, Allow wins. So Disallow: /admin/ with Allow: /admin/help/ keeps the help pages crawlable whatever order you write them in. A crawler also follows only one group: Googlebot obeys a Googlebot group and ignores the * group entirely.

    Blocking is not the same as noindex

    A URL blocked in robots.txt can still appear in search results, usually as a bare link with no description, because other pages link to it. Worse, if you add a noindex tag to a page and also block it, the crawler never fetches the page and never sees the tag. To remove a page from search, allow crawling and use noindex; block it only once it has dropped out.

    Meta Tag Generator →

    Mistakes that block more than you meant

    Rules are prefixes: Disallow: /blog also blocks /blog-tips/ and /blogger.html; write /blog/ if you mean the folder. Disallow: / on its own blocks the whole site, a classic leftover from a staging server. Blocking CSS and JavaScript folders stops search engines from rendering your pages properly. And paths are case-sensitive: /Admin/ does not cover /admin/. To see how a particular URL is treated, enter it in the test box under the output: it names the group and the rule that decide.

    AI crawlers and control tokens

    Several AI companies publish the user-agent names they use to collect training data and document robots.txt as the way to opt out. Some names are not separate crawlers but control tokens: Google-Extended and Applebot-Extended never fetch pages; they only tell the company whether content its search crawler has already collected may be used to train its AI models (for Google, also to ground Gemini answers). Blocking them does not affect your search ranking, but it does not keep your pages out of AI features inside search either: Google’s AI Overviews and AI Mode run on Googlebot and are controlled with nosnippet or noindex, not with Google-Extended. Names change and new ones appear, so treat the preset as a starting point and check each provider’s current documentation.

    Robots.txt questions

    No. It only stops crawling. If other sites link to the URL it can stay indexed, shown without a description. To remove it, let it be crawled with a noindex meta tag or X-Robots-Tag header, or take it down or put it behind a login.
    New
    UTM Builder Build UTM-tagged campaign links in seconds. Keeps existing parameters and #anchors, encodes values correctly and warns about capital letters and spaces.
    Popular Today
    Keyword Density Checker Free keyword density checker: paste text or a URL to see how often each word and phrase appears, with percentages for one, two and three-word terms.

    Other SEO tools

    Keyword Density Checker Free keyword density checker: paste text or a URL to see how often each word and phrase appears, with percentages for one, two and three-word terms.
    Slug Generator Turn any headline into a clean URL slug. German ü becomes ue, Turkish ü becomes u, Cyrillic is transliterated — you pick the language, line by line.
    Meta Tag Generator Fill in a title, description, URL and image to generate HTML meta tags, Open Graph and Twitter Card markup, and see how the page will look in Google results, on Facebook and on X before you publish.
    SERP Snippet Preview See how your title and meta description look in search results on desktop and mobile. Pixel-based cut-off, character and pixel counters, live preview.
    UTM Builder Build UTM-tagged campaign links in seconds. Keeps existing parameters and #anchors, encodes values correctly and warns about capital letters and spaces.

    Other Tools You May Like

    Word to PDF Convert Word to PDF for free: upload a DOC, DOCX, ODT or RTF file and download a properly formatted PDF in seconds - no registration or watermark.
    PDF to Word Convert PDF to Word for free: upload a PDF and download an editable DOCX in seconds - no registration, no watermark and nothing to install.
    MD5 Hash Generator Free MD5 hash generator: turn any text or file into an MD5 checksum in your browser - useful for integrity checks and quick fingerprinting.
    Base64 Encoder & Decoder Free Base64 encoder and decoder: convert text or files to Base64 and back, entirely in your browser - nothing is uploaded to a server.
    Word Counter Free word counter: paste any text to count words, characters, sentences and paragraphs, and see the estimated reading time - nothing leaves your browser.