Robots.txt generator

What is a robots.txt file?

A robots.txt file sits at the root of a domain and tells crawlers which parts of the site they may request. It is a plain text file of groups: one or more User-agent lines naming a crawler, followed by Allow and Disallow lines naming paths. That is the whole format.

For twenty-five years it was a convention rather than a standard, which is why so much folklore surrounds it. That changed in September 2022, when it was published as RFC 9309, the Robots Exclusion Protocol. Having an actual specification means questions that used to be answered by opinion now have a citation, and this generator follows it.

The single most useful thing to understand is that robots.txt controls crawling, not indexing, and it is a request rather than a fence. Well-behaved crawlers honour it. Nothing stops a badly behaved one, and the file is public, so it is not a place to list the paths you most want kept quiet.

How to use it

1. Fill in the crawler and the paths. Leave the crawler as an asterisk to address every crawler, or name one to write rules that apply to it alone. Disallow takes one path per line, and Allow is for carve-outs inside a blocked path. The presets give you a starting point for allowing everything, blocking everything, or blocking a handful of paths.
2. Edit or paste directly into the output box. The generated file is editable, and you can replace it wholesale with a robots.txt you already have. Everything below the box reads whatever is in it, so the tester works just as well on a file this tool did not write.
3. Test a real URL against it. Enter a path and a crawler name and the tool says allowed or blocked, then names the rule that decided, its length in octets, and which group matched. That is the part worth checking, because the deciding rule is often not the one you expect.

What the standard actually says, and what it does not

RFC 9309 defines exactly three directives: User-agent, Allow and Disallow. That is the complete list. Crawl-delay is not in the standard, and neither is Host. Sitemap is not part of the protocol either, though the specification does mention it as an example of a record crawlers may choose to interpret, which is why this tool emits it and labels it as an extension rather than a directive.

This tool therefore has no Crawl-delay field, on purpose. Google has stated plainly that it ignores the line, and it is not in the standard, so a generator that offers it hands you something that looks like a rate limit and is not one. If your file already contains it, paste the file in and the tool will list it under what is in the file, marked as not in the standard. Nothing is hidden, but nothing is invented either.

The reverse is also worth knowing, and it surprises people more. The asterisk and dollar wildcards are widely believed to be a Google extension. They are in the standard: an asterisk matches zero or more of any character, and a dollar sign designates the end of the match pattern. So the directives everyone treats as standard mostly are not, and the syntax everyone treats as non-standard is.

The rule that decides is the longest one, not the first

When several rules match a URL, the specification is explicit: the most specific match must be used, and the most specific match is the one with the most octets. Order in the file is irrelevant. So Disallow slash a followed by Allow slash a slash b permits /a/b/c, because the Allow is four octets against the Disallow's two. This is the single most common misunderstanding about robots.txt, and it is why the tester shows the deciding rule and its length rather than just a verdict.

If an Allow and a Disallow of exactly the same length both match, the standard says the Allow should win. The tool reports that case explicitly when it happens, since two rules of equal length pointing opposite ways is usually a sign the file says something its author did not intend.

Two more rules catch people out, and they point in opposite directions. Crawler names are matched case-insensitively, so Googlebot and googlebot are the same group. Paths are matched case-sensitively, so Disallow slash Admin does nothing to /admin. One file, two different rules about capitals. Related: when more than one group names the same crawler, the standard requires their rules to be combined into a single group rather than the first one winning.

Finally, a rule that matches nothing is not a rule that blocks. If no rule matches, the URL is allowed, and an empty Disallow line means exactly that: allow everything. A group with no rules in it says nothing at all, which is why this generator writes an explicit empty Disallow when you ask it to allow everything.

Honest limits

The most important limitation is not about this tool but about the file. Disallow stops a crawler fetching a page; it does not remove that page from search results. A URL that is linked from elsewhere can still be listed, without a description, because the crawler was told not to look at it. If you want a page out of an index, the page has to be crawlable and carry a noindex instruction, or be removed. Blocking it in robots.txt is the one thing that guarantees the instruction is never seen.

Nor is it access control. The file is world-readable at a predictable address, so listing a sensitive path advertises it. Anything that genuinely must not be reached needs authentication, not a polite request.

There is one behaviour worth knowing before you touch a live file, and almost nobody does. If robots.txt returns a 404 the crawler may access anything, which is the forgiving case. If it returns a 5xx server error, the standard says the crawler must assume complete disallow. So a robots.txt that is fine but sits on a server having a bad hour reads, to a conforming crawler, as though the entire site were blocked. That is a deployment failure mode, not a syntax one, and no validator can warn you about it.

This tool also checks one file in isolation. It cannot tell you whether the paths you typed exist, whether your server actually serves the file at the root, or how a specific crawler behaves in practice, since the standard leaves plenty of room and crawlers use it. It implements what the RFC specifies, and where a crawler goes beyond that, the tool says so rather than guessing.

Why is it free?

Because it is text in and text out. There is no server involved and nothing to download: the parser, the matcher and the generator all run in your browser, so nothing you type or paste is uploaded, logged or stored anywhere. Your robots.txt is a public file, but the rest of your site structure showing up in the box beside it is not, and it does not leave the page.

So there is no account, no sign-up and nothing held back. The engine and the checks that verify it sit in the repository next to the page, including an exhaustive sweep of 248,248 pattern and path combinations against an independent matcher, with no disagreements.