Robots.txt Generator
Create robots.txt files for crawler control
Robots.txt Generator tool
Robots.txt Generator: key facts
- What it does
- Create robots.txt files for crawler control
- Category
- SEO Tools
- Cost
- Free, with no account, sign-up, or install.
- Your data
- Runs entirely in your browser — the files and text you enter are never uploaded to a server.
- Last reviewed
- . Report an incorrect result.
What this tool is for
A robots.txt file tells search engine crawlers which parts of your site they may and may not visit, and a single typo can accidentally hide your whole site or expose pages you meant to block. This generator builds a correct robots.txt visually, so you set rules without memorizing the syntax.
Site owners and developers use it when launching a site, blocking admin or staging areas, or pointing crawlers at a sitemap. Presets cover the common cases — allow everything, block everything, or a WordPress-tuned setup — while custom mode lets you define rules per user-agent, reorder them, and add crawl delays.
How to use the Robots.txt Generator
- Start from a preset (Allow All, Block All, or WordPress) or build from scratch.
- For each user-agent, add Allow or Disallow rules with their paths, reordering them as needed.
- Add more user-agents if you need different rules for specific crawlers.
- Add your sitemap URL, generate the file, and copy it into your site's root as robots.txt.
User-agent groups, rules, and the sitemap line
The generator assembles a robots.txt from user-agent blocks, each containing allow and disallow rules, and appends a sitemap reference if you supply one. The file lives at the root of a domain and nowhere else — a robots.txt in a subdirectory is simply ignored.
Rules are grouped under the user-agent they apply to, and a crawler obeys the most specific group matching its own name, ignoring the others entirely. That is the detail most often got wrong: a crawler that finds a group naming it will not also read the wildcard group, so shared rules must be repeated rather than assumed.
Path matching is prefix-based from the root, so a disallow on a directory covers everything beneath it. Order within a group does not decide precedence — the longest matching rule wins, and allow beats disallow when both match at the same length.
- Disallow with an empty value permits everything; Disallow: / blocks the whole site.
- A group naming a specific crawler is read by that crawler instead of the wildcard group, not in addition to it.
- The sitemap directive is independent of user-agent groups and applies to the whole file.
What the Robots.txt Generator gets right
- Presets for Allow All, Block All, and WordPress cover the most common setups, while custom mode handles anything specific.
- It supports multiple user-agents with reorderable rules and optional crawl-delay directives, matching the full robots.txt spec.
- It appends your sitemap reference, the line crawlers look for to discover your URLs.
- It generates the file locally with no account; you simply copy it to your site root.
Blocking crawling is not blocking indexing
This is the misunderstanding that causes real damage. Disallowing a URL stops a crawler fetching it — it does not stop the URL appearing in results, because Google can index a page it has never fetched on the strength of links pointing at it. The result is a listing with no description.
Worse, the two mechanisms conflict. A noindex meta tag only works if the crawler is allowed to fetch the page and read it. Disallowing a page you have marked noindex means the directive is never seen, so the page can stay indexed indefinitely. To remove a page from results, allow crawling and use noindex; to keep something genuinely private, require authentication.
Two further cautions. robots.txt is public, so listing sensitive paths advertises exactly what you hoped to hide. And it is advisory: well-behaved crawlers honour it, while scrapers and malicious bots simply ignore it. Blocking CSS and JavaScript is also counterproductive, since Google renders pages and needs those assets to see the layout as a user would.
Frequently Asked Questions
What is a robots.txt file for?
It is a plain-text file at your site's root that tells search engine crawlers which paths they may or may not request. It guides crawling, but note it is not a security measure — disallowed pages can still be accessed directly and may appear in results if linked elsewhere.
What is the difference between Allow and Disallow?
Disallow tells a crawler not to request a path, while Allow explicitly permits one — useful for carving out an exception inside a disallowed directory. The generator lets you mix both per user-agent and order them so the right rule wins.
Where do I put the generated file?
Save it as robots.txt in the root of your domain, so it is reachable at yoursite.com/robots.txt. Crawlers look for it there specifically; placed anywhere else, it will be ignored.
Should I block pages with robots.txt or noindex?
To keep a page out of search results, a noindex meta tag is more reliable, because robots.txt only controls crawling, not indexing. Use robots.txt to manage crawler load and access; use noindex when the goal is to keep something out of the index.
