robots.txt Generator

What Is a robots.txt File?

A robots.txt file is a plain-text file that lives in the root directory of your website and tells automated crawlers which parts of your site they may or may not request. It follows the Robots Exclusion Protocol, a long-standing convention that search engines like Google and Bing voluntarily obey. The file is organized into groups, each beginning with one or more User-agent lines that name the crawler the rules apply to, followed by Disallow and Allow directives that define the paths in question. A single robots.txt can contain many groups, so you can give Googlebot one set of instructions and a generic * wildcard another.

It is important to understand what robots.txt does and does not do. It is a crawl-control mechanism, not a security or privacy barrier. Well-behaved bots respect it, but malicious scrapers can ignore it entirely, and any path you list is publicly visible to anyone who reads the file. The protocol also supports a non-standard Crawl-delay directive that asks bots to wait a number of seconds between requests, plus a Sitemap line that points crawlers to your XML sitemap and an occasionally used Host directive that some search engines read to identify a preferred domain.

How to Use This robots.txt Generator

Start with the first user-agent rule. Pick a crawler from the User-agent dropdown — choose All robots (*) to target every bot, or a specific one such as Googlebot or Bingbot. For each rule, set the Disallow/Allow directive and type the path (for example /admin/) in the path field. Click + Add Path to stack multiple directives under the same user-agent, or + Crawl-delay to insert a crawl-delay value of 1–60 seconds. Use + Add User-agent Rule to create separate groups for different bots, and the ✕ button to remove any rule or path you no longer need.

Below the rules, fill in the Sitemap URL field to append a Sitemap: line, and the optional Host field if you want a host directive. Tick Block AI crawlers to automatically add blocking groups for GPTBot, CCBot, Google-Extended, and anthropic-ai. The output regenerates live in the right-hand pane as you type — when it looks right, press Copy and save the text as robots.txt in your site root.

Real-World Use Cases

• Hiding admin and staging areas — Disallow paths like /wp-admin/ or /staging/ so search engines stop wasting crawl budget on pages no visitor should reach from search.
• Pointing crawlers to your sitemap — Adding a Sitemap: line helps search engines discover all of your important URLs quickly, which is especially useful for large or newly launched sites.
• Blocking AI training crawlers — Publishers who do not want their content used to train large language models can disallow GPTBot, CCBot, Google-Extended, and anthropic-ai in one click.
• Rate-limiting aggressive bots — Applying a Crawl-delay to a specific user-agent like Bingbot or YandexBot can ease load on a server that struggles under heavy crawling.
• Per-bot rules — Giving Googlebot full access while restricting lesser-known crawlers lets you keep your best traffic source happy without inviting every scraper in.

Tips for Writing a robots.txt File

• Paths are matched as prefixes and are case-sensitive, so /Blog/ and /blog/ are treated as different directories.
• Use Disallow: / to block an entire site for a given user-agent, and an empty Disallow: (no path) to allow everything.
• When a parent directory is disallowed, add an Allow rule for a specific sub-path to carve out an exception — Google honors the more specific match.
• Never rely on robots.txt to hide sensitive content, because the file is public and disallowed pages can still be indexed if other sites link to them.
• Validate the final file in Google Search Console's robots.txt report after uploading, and remember it must be reachable at exactly https://yourdomain.com/robots.txt.

Features

• Multiple user-agent groups — add as many rules as you need, each targeting a different crawler or the * wildcard.
• Common bot presets — pick from Googlebot, Bingbot, Yahoo Slurp, DuckDuckBot, Baiduspider, YandexBot, and more without typing names by hand.
• Allow and Disallow directives — stack multiple paths per rule with one-click add and remove controls.
• Crawl-delay support — insert a delay value between 1 and 60 seconds for any user-agent.
• Sitemap and Host lines — append both with dedicated fields.
• One-click AI-crawler blocking — disallow GPTBot, CCBot, Google-Extended, and anthropic-ai instantly.
• Live output and copy — the file regenerates as you edit, ready to copy in a single click.
• 100% client-side — your data never leaves your browser.

Frequently Asked Questions

Where do I put my robots.txt file?

It must live in the root directory of your domain so it is reachable at https://yourdomain.com/robots.txt. Crawlers only look in that exact location, so placing it in a subfolder will have no effect.

Does robots.txt prevent a page from being indexed?

No. It only controls crawling. A disallowed URL can still appear in search results if other pages link to it, because the crawler sees the link without reading the page. To keep a page out of the index entirely, use a <meta name="robots" content="noindex"> tag or an X-Robots-Tag HTTP header, and do not also disallow it — the crawler must be able to read the page to see the noindex instruction.

What does blocking AI crawlers actually do?

It adds User-agent groups for GPTBot, CCBot, Google-Extended, and anthropic-ai with a Disallow: / rule, asking those crawlers not to access your site for AI training or generative-search purposes. These crawlers state that they honor robots.txt, but the directive is advisory and cannot force compliance from bots that ignore the protocol.

Do all crawlers respect the Crawl-delay directive?

No. Crawl-delay is a non-standard extension. Bing and Yandex historically honored it, but Google ignores it and instead manages crawl rate through Search Console settings. Treat it as a polite request rather than a guarantee.

Can I have more than one robots.txt file?

You should have only one per host, located at the root. Each subdomain is treated separately, so blog.example.com and shop.example.com each need their own file. Within that single file you can define as many user-agent groups as you like.

Is my data sent anywhere when I use this generator?

No. The entire file is assembled in JavaScript inside your browser, so nothing you type is uploaded or stored on a server. You can use the tool offline once the page has loaded, and the generated text exists only in your browser until you copy it.