robots.txt Generator

Web

Everything runs locally in your browser — nothing is uploaded

Policy

Crawler names to block (comma or space separated)

Paths to disallow (space or comma separated)

Sitemap URL (leave empty to omit)

Crawl-delay seconds (0 omits the line)

Add English comments explaining each section

Output

Press Ctrl/⌘ + Enter to run · Press Esc to clear

Generate a robots.txt from one of a few common policies — a few minutes of filling in fields. ⚠️ First, what robots.txt is: a voluntary convention, not access control. Well-behaved crawlers (Google, Bing and the other large search engines) read it, and any crawler may ignore it — anything that genuinely needs protecting belongs behind authentication, not this file. It also cannot remove pages that are already indexed (that requires a removal request in the search engine’s webmaster tools). How the three policies differ: - Allow everything — a single empty `Disallow:` value, explicitly meaning “nothing is disallowed” - Block AI training crawlers — lists GPTBot, ClaudeBot, CCBot, Google-Extended and similar model-training crawlers, disallowed site-wide - Block everything — one `Disallow: /` ⚠️ The AI crawler list has a shelf life: new players appear every few months, so the generated file carries a comment stating its review date, and this page claims only “this batch is blocked”, never “all of them”. A more complete list means maintaining your own — use the custom list field for the ones you care about. ⚠️ All comments in the file are English — it is published on your site, for crawlers and peers worldwide to read. ⚠️ One more: Google explicitly ignores `Crawl-delay` (it uses its own webmaster-tools setting), while Yandex and Bing honour it. Setting it is not a guarantee of rate limiting.

Features

Four policies: allow all, block AI training crawlers, block all, custom list
Custom list takes any crawler names (comma, space, ideographic comma and full-width comma all work)
Disallow paths get a leading slash and a directory trailing slash automatically, and are de-duplicated
Optional Sitemap line and Crawl-delay
All comments in the generated file are English (it is published on your site)
Changing any option re-renders immediately — no button to press
No network, nothing uploaded — generated entirely on your device

How to use

Pick a policy
Fill in disallow paths, Sitemap and Crawl-delay as needed
Copy the result and save it as robots.txt in your site root

FAQ

If I block the AI crawlers, do they actually stop?: Only the ones that play by the rules. OpenAI, Anthropic, Google and Apple publish their crawler tokens and state that they honour robots.txt, so blocking them does stop those fetches (`Google-Extended` and `Applebot-Extended` express “you may crawl, but do not train on it” — those two exist specifically to be switched off). But anyone may ignore the file. A crawler with a changed UA, or a scraping service that does not care, cannot be stopped from robots.txt at all. Real protection means authentication, rate limiting, or not publishing the content. And the list itself ages: new training crawlers appear every few months, which is why the generated file states its review date.
Is there a difference between `Disallow: /admin` and `Disallow: /admin/`?: Yes, and it is frequently missed. `Disallow: /admin` matches every path beginning with `/admin` — including `/administrator` and `/admin-panel.html`, which are often not what you meant. The trailing slash makes it refer to that directory only. So when you write a bare directory name (no dot, no asterisk, no `$`), this tool adds the trailing slash for you; filenames like `/admin.html` and patterns like `/a/*.pdf$` are kept exactly as written.
Why are the generated comments in English?: Because this file is published on your site — anyone fetching `/robots.txt` (including the operators of every crawler) will read it. Chinese-only comments are unfriendly to those operators, and the point of the comments is precisely to explain why each section says what it does. Only the crawler names and paths are your content; the rest is written by the generator, so that part is fixed to English. As a side effect, the output does not depend on the interface language — switch to Japanese and the result is identical.

Related tools

.gitignore Generator

Web

Generate a .gitignore from language, framework, editor and OS templates. Sections are merged and duplicates removed. Runs entirely in your browser.

HTTP Header Formatter

Web

Turn captured or curl-pasted HTTP headers into a table, convert byte sizes and charsets, and see which security headers a response is missing.

URL Parser

Web

Parse a URL into scheme, host, port, path, query parameters and fragment, with each parameter shown raw and decoded. Runs entirely in your browser.

User-Agent Parser

Web

Parse a user agent into browser, version, OS, device type and engine, and see which rules matched. Pattern matching only. Runs entirely in your browser.

HTTP Status Codes

Web

Look up an HTTP status code for its meaning, 1xx–5xx class, typical uses and common traps. Runs entirely in your browser.

MIME Types

Web

Look up a MIME type from an extension, or the default extension for a media type. Common types only. Runs entirely in your browser.

← Back to all tools