What is a Robots.txt Generator?
A robots.txt file sits at the root of a domain and tells crawlers which paths they may request. This generator writes one from a few choices: allow everything, block everything, exclude specific paths, and optionally name a sitemap.
The syntax is small. A `User-agent` line starts a group, `Allow` and `Disallow` lines set rules for that group, and matching is by prefix — `Disallow: /admin/` blocks `/admin/users` but not `/administrator`. Google also understands `*` as a wildcard and `$` as an end anchor, though the original 1994 specification did not.
What robots.txt is not is access control. It is a request, honoured by well-behaved crawlers and ignored by scrapers, so anything genuinely private must sit behind authentication. It also does not remove a page from the index: a blocked URL can still be listed if other pages link to it. For that you need a `noindex` directive, which the crawler can only see if it is allowed to fetch the page.
How to write a robots.txt
- Choose allow everything unless the whole site is private. That writes `Allow: /` for all user agents.
- Add paths to exclude, one per line, such as `/admin/` or `/cart/`. A leading slash is added automatically.
- Paste the absolute sitemap URL. One `Sitemap:` line is enough; the file can list several.
- Set a crawl delay only if a smaller crawler is hammering the server — Google and Bing ignore the directive.
- Copy the output and save it as `robots.txt` at the domain root, so it answers at `https://example.com/robots.txt`.
- Fetch it in a browser and in Search Console's robots.txt report to confirm it parses without errors.
Robots.txt Generator limits and rules
| Item | Limit |
|---|---|
| Location | Domain root only — `/robots.txt`, one per scheme and host |
| Matching | Prefix by default; `*` wildcard and `$` end anchor on Google |
| Rule precedence | The most specific match wins; `Allow` wins a tie on Google |
| Crawl delay | Honoured by Yandex and some crawlers; ignored by Google and Bing |
| File size | Google reads the first 500 KiB |
| Not access control | Scrapers ignore it; use authentication for private data |
| Not deindexing | Use a `noindex` meta tag or header, and allow the fetch |
| Subdomains | Each host needs its own file — `blog.example.com` is separate |
When to use it
- Blocking an admin panel, a cart or a search-results page from being crawled.
- Keeping a staging copy out of the index before launch, ideally alongside HTTP authentication.
- Pointing crawlers at a sitemap so new pages are discovered sooner.
- Telling crawlers to stay out of an API or a JSON endpoint that should never appear in search.
- Slowing a crawler that is consuming too much bandwidth on a small server.
Is it private?
Everything runs inside your browser. Your text and files are never uploaded, never stored and never seen by anyone else — including me. You can use the tool with the network tab open and watch no request leave the page. Closing the tab erases the input.
Frequently asked questions
Not reliably. `Disallow` stops the crawl, not the indexing — a blocked URL can still be listed if it is linked from elsewhere, usually with no snippet. To remove a page, allow the crawl and add a `noindex` meta tag or `X-Robots-Tag` header, then request removal in Search Console.
No. It is a convention, not a security control: a scraper that ignores it will fetch everything, and the file itself advertises where the interesting paths are. Anything private belongs behind authentication or outside the web root.
No. Google renders pages to evaluate them, and blocking the stylesheets and scripts it needs makes the page look broken. The old advice to disallow `/js/` and `/css/` came from a time when rendering was not part of indexing.
Because Google and Bing never implemented it. Google recommends controlling crawl rate in Search Console instead, and Bing uses its own crawl settings. The directive is still useful for Yandex and some smaller crawlers, which is why it is offered as an option.
Not strictly — without one, crawlers assume they may fetch everything. It becomes useful when you want to exclude paths, point at a sitemap, or stop a specific crawler. An empty file and a missing file mean the same thing.
No. The file is assembled in the page from your choices and copied from there. Nothing is saved, so you can draft rules for a site that is not public yet.
Related tools
Written and maintained by Md. Nur Hossain Repon.