NHRepon
Home
Services
All ServicesWeb Application DevelopmentMobile Application DevelopmentDigital Marketing and SEO
ProjectsToolsProductsPostsAboutContactLogin
HomeServicesProjectsToolsProductsPostsAboutContact
  • Home
  • Projects
  • Posts
Home/Tools/SEO Tools/Robots.txt Generator
All Tools

Robots.txt Generator

Write a robots.txt with an allow-all or block-all rule, per-path exclusions, a sitemap line and an optional crawl delay.

Crawler access

Each line becomes a Disallow rule. A leading slash is added if you leave it out.

robots.txt
User-agent: *
Allow: /

Save it as robots.txt at the domain root. Crawl-delay is honoured by Yandex and some smaller crawlers; Google and Bing ignore it and use Search Console rate limits instead.

What is a Robots.txt Generator?

A robots.txt file sits at the root of a domain and tells crawlers which paths they may request. This generator writes one from a few choices: allow everything, block everything, exclude specific paths, and optionally name a sitemap.

The syntax is small. A `User-agent` line starts a group, `Allow` and `Disallow` lines set rules for that group, and matching is by prefix — `Disallow: /admin/` blocks `/admin/users` but not `/administrator`. Google also understands `*` as a wildcard and `$` as an end anchor, though the original 1994 specification did not.

What robots.txt is not is access control. It is a request, honoured by well-behaved crawlers and ignored by scrapers, so anything genuinely private must sit behind authentication. It also does not remove a page from the index: a blocked URL can still be listed if other pages link to it. For that you need a `noindex` directive, which the crawler can only see if it is allowed to fetch the page.

How to write a robots.txt

  1. Choose allow everything unless the whole site is private. That writes `Allow: /` for all user agents.
  2. Add paths to exclude, one per line, such as `/admin/` or `/cart/`. A leading slash is added automatically.
  3. Paste the absolute sitemap URL. One `Sitemap:` line is enough; the file can list several.
  4. Set a crawl delay only if a smaller crawler is hammering the server — Google and Bing ignore the directive.
  5. Copy the output and save it as `robots.txt` at the domain root, so it answers at `https://example.com/robots.txt`.
  6. Fetch it in a browser and in Search Console's robots.txt report to confirm it parses without errors.

Robots.txt Generator limits and rules

Limits, caps and standards referenced by this tool
ItemLimit
LocationDomain root only — `/robots.txt`, one per scheme and host
MatchingPrefix by default; `*` wildcard and `$` end anchor on Google
Rule precedenceThe most specific match wins; `Allow` wins a tie on Google
Crawl delayHonoured by Yandex and some crawlers; ignored by Google and Bing
File sizeGoogle reads the first 500 KiB
Not access controlScrapers ignore it; use authentication for private data
Not deindexingUse a `noindex` meta tag or header, and allow the fetch
SubdomainsEach host needs its own file — `blog.example.com` is separate

When to use it

  • Blocking an admin panel, a cart or a search-results page from being crawled.
  • Keeping a staging copy out of the index before launch, ideally alongside HTTP authentication.
  • Pointing crawlers at a sitemap so new pages are discovered sooner.
  • Telling crawlers to stay out of an API or a JSON endpoint that should never appear in search.
  • Slowing a crawler that is consuming too much bandwidth on a small server.

Is it private?

Everything runs inside your browser. Your text and files are never uploaded, never stored and never seen by anyone else — including me. You can use the tool with the network tab open and watch no request leave the page. Closing the tab erases the input.

Frequently asked questions

Not reliably. `Disallow` stops the crawl, not the indexing — a blocked URL can still be listed if it is linked from elsewhere, usually with no snippet. To remove a page, allow the crawl and add a `noindex` meta tag or `X-Robots-Tag` header, then request removal in Search Console.

No. It is a convention, not a security control: a scraper that ignores it will fetch everything, and the file itself advertises where the interesting paths are. Anything private belongs behind authentication or outside the web root.

No. Google renders pages to evaluate them, and blocking the stylesheets and scripts it needs makes the page look broken. The old advice to disallow `/js/` and `/css/` came from a time when rendering was not part of indexing.

Because Google and Bing never implemented it. Google recommends controlling crawl rate in Search Console instead, and Bing uses its own crawl settings. The directive is still useful for Yandex and some smaller crawlers, which is why it is offered as an option.

Not strictly — without one, crawlers assume they may fetch everything. It becomes useful when you want to exclude paths, point at a sitemap, or stop a specific crawler. An empty file and a missing file mean the same thing.

No. The file is assembled in the page from your choices and copied from there. Nothing is saved, so you can draft rules for a site that is not public yet.

Related tools

  • robots.txt Tester
  • XML Sitemap Generator
  • Meta Tag Generator
  • Disavow File Generator
  • SEO Audit

Written and maintained by Md. Nur Hossain Repon.

NHRepon


Md. Nur Hossain Repon expert in NextJS, React, React Native and Flutter. He is a full-stack web and mobile application developer with 5+ years of experience and a passion for building innovative and robust application. Also, he has a strong background in developing high-quality web and mobile applications using the latest technologies and frameworks.
Let's build something amazing together 🚀

Quick Links


  • About
  • Projects
  • Services
  • Contact
  • Posts
  • Products
  • Privacy Policy
  • Terms of Service
  • Tools
  • Smart Life Hubs

Newsletter


We respect your privacy. Unsubscribe anytime.

Connect


© 2026 NHRepon. All rights reserved.

Build v1.5.4+56.73c4a90