robots.txt Generator & Tester

What Is a robots.txt File?

A robots.txt file is a plain-text file that lives in the root directory of your website and tells automated crawlers which parts of your site they may or may not request. It follows the Robots Exclusion Protocol, a long-standing convention that search engines like Google and Bing voluntarily obey. The file is organized into groups, each beginning with one or more User-agent lines that name the crawler the rules apply to, followed by Disallow and Allow directives that define the paths in question. A single robots.txt can contain many groups, so you can give Googlebot one set of instructions and a generic * wildcard another.

It is important to understand what robots.txt does and does not do. It is a crawl-control mechanism, not a security or privacy barrier. Well-behaved bots respect it, but malicious scrapers can ignore it entirely, and any path you list is publicly visible to anyone who reads the file. Since 2022 the format is an official internet standard (RFC 9309). Besides User-agent, Allow and Disallow, most crawlers read a Sitemap line that points to your XML sitemap. The non-standard Crawl-delay is read by Bing and Yandex but ignored by Google, and the old Host directive is no longer used by any major search engine.

How to Use This robots.txt Generator

The quickest start is a template: Allow all, WordPress (blocks /wp-admin/ but keeps admin-ajax.php reachable, plus internal search pages), WooCommerce shop (adds cart, checkout, account and add-to-cart/sort URLs), Block AI training, Block all AI bots or Staging site. Every template is just a starting point — change it as you like.

To build rules by hand, pick a crawler from the User-agent dropdown — All robots (*), a specific bot such as Googlebot or Bingbot, or Custom… to type any name. Set Disallow or Allow and type the path (for example /admin/); leave it empty to allow everything. + Add Path stacks more lines under the same user-agent, + Crawl-delay adds a delay of 1–60 seconds, and + Add User-agent Rule creates a separate group for another bot. Choose an AI crawlers policy and add one or more Sitemap URLs (separate several with a space). The file updates live — press Copy or Download and upload it as robots.txt to your site root.

How the robots.txt Tester Works

The tester below the generator checks any robots.txt — paste your live file or test the one you just built. Enter a URL or path and a crawler name, and it tells you whether that crawler may fetch the URL, which group applied and which line decided. It follows the rules Google documents and RFC 9309 describes:

• One group per crawler. A crawler obeys only the group with the most specific matching user-agent. If Googlebot has its own group, it ignores the * group completely. Crawlers like Googlebot-Image fall back to the Googlebot group when they have none of their own.
• Longest match wins. Within the group, the rule with the longest matching path decides. Allow: /fish/salmon beats Disallow: /fish for /fish/salmon.html. If an Allow and a Disallow rule are equally long, Allow wins.
• Wildcards. * matches any sequence of characters and $ marks the end of the URL — Disallow: /*.php$ blocks /index.php but not /index.php?id=1.
• Problems are listed by line: rules before any User-agent, missing colons, misspelled directives, unsupported Noindex, relative sitemap URLs, paths without a leading slash and files over Google’s 500 KiB limit.

The overview below the result shows the same URL for 14 common crawlers at once — handy for seeing at a glance which search engines and AI bots can reach a page.

Real-World Use Cases

• Hiding admin and staging areas — Disallow paths like /wp-admin/ or /staging/ so search engines stop wasting crawl budget on pages no visitor should reach from search.
• Pointing crawlers to your sitemap — Adding a Sitemap: line helps search engines discover all of your important URLs quickly, which is especially useful for large or newly launched sites.
• Blocking AI training but staying visible in AI search — Disallow training crawlers like GPTBot, ClaudeBot, Google-Extended and CCBot while still letting OAI-SearchBot, Claude-SearchBot and PerplexityBot index your pages, so AI assistants can cite and link to you.
• Debugging a ranking drop — Paste your live robots.txt into the tester to check that an important page is not accidentally blocked for Googlebot.
• Rate-limiting aggressive bots — Applying a Crawl-delay to a specific user-agent like Bingbot or YandexBot can ease load on a server that struggles under heavy crawling.
• Per-bot rules — Giving Googlebot full access while restricting lesser-known crawlers lets you keep your best traffic source happy without inviting every scraper in.

Tips for Writing a robots.txt File

• Paths are matched as prefixes and are case-sensitive, so /Blog/ and /blog/ are treated as different directories.
• Use Disallow: / to block an entire site for a given user-agent, and an empty Disallow: (no path) to allow everything.
• When a parent directory is disallowed, add an Allow rule for a specific sub-path to carve out an exception — the longer (more specific) rule wins.
• Do not block CSS and JavaScript files. Google renders pages like a browser and needs them to understand your layout.
• If you give a bot its own group, repeat every rule it should follow — it no longer reads the * group.
• Never rely on robots.txt to hide sensitive content, because the file is public and disallowed pages can still be indexed if other sites link to them.
• Check the final file with the tester above and in Google Search Console’s robots.txt report after uploading. It must be reachable at exactly https://yourdomain.com/robots.txt.

Features

• Templates — Allow all, WordPress, WooCommerce, Block AI training, Block all AI bots and Staging in one click.
• Multiple user-agent groups — add as many rules as you need, for 25 common crawlers or any custom name.
• Current AI crawler list — training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta, Bytespider, Amazonbot …), AI search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and user fetchers (ChatGPT-User, Claude-User, Perplexity-User), blockable separately.
• Allow, Disallow and Crawl-delay — stack multiple paths per rule with one-click add and remove controls.
• Tester and validator — check any URL against any robots.txt with Google’s matching rules, see the deciding line, compare 14 crawlers at once and get a list of problems.
• Copy or download — the file regenerates as you edit, ready to copy or save as robots.txt.
• 100% client-side — your data never leaves your browser.

Frequently Asked Questions

Where do I put my robots.txt file?

It must live in the root directory of your domain so it is reachable at https://yourdomain.com/robots.txt. Crawlers only look in that exact location, so placing it in a subfolder will have no effect.

Does robots.txt prevent a page from being indexed?

No. It only controls crawling. A disallowed URL can still appear in search results if other pages link to it, because the crawler sees the link without reading the page. To keep a page out of the index entirely, use a <meta name="robots" content="noindex"> tag or an X-Robots-Tag HTTP header, and do not also disallow it — the crawler must be able to read the page to see the noindex instruction.

What does blocking AI crawlers actually do?

It adds one group listing the AI crawlers’ user-agents with a Disallow: / rule. Block AI training covers crawlers that collect content for model training (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, meta-externalagent, Bytespider, Amazonbot and others) but keeps AI search crawlers allowed, so ChatGPT, Claude or Perplexity can still find and cite your pages. Block all AI bots also blocks the search crawlers and the user fetchers that load a page when a person asks an assistant about it. The major AI companies state that their crawlers honor robots.txt, but the file is advisory and cannot stop bots that ignore it.

Does blocking Google-Extended remove my site from Google Search?

No. Google-Extended is not a separate crawler but a control token: it tells Google not to use your content for training and grounding its Gemini models. Googlebot keeps crawling and indexing your site for Google Search as usual. Applebot-Extended works the same way for Apple.

What happens when Allow and Disallow rules conflict?

Google and RFC 9309 use the most specific rule — the one with the longest matching path. With Disallow: /shop/ and Allow: /shop/sale/, the URL /shop/sale/shoes is allowed and /shop/cart is blocked. If both rules are the same length, Allow wins. Other crawlers may resolve conflicts differently, so avoid ambiguous rules where you can. The tester on this page shows which line decides.

Do all crawlers respect the Crawl-delay directive?

No. Crawl-delay is a non-standard extension. Bing and Yandex historically honored it, but Google ignores it and instead manages crawl rate through Search Console settings. Treat it as a polite request rather than a guarantee.

Can I have more than one robots.txt file?

You should have only one per host, located at the root. Each subdomain is treated separately, so blog.example.com and shop.example.com each need their own file. Within that single file you can define as many user-agent groups as you like.

Is my data sent anywhere when I use this generator?

No. The entire file is assembled in JavaScript inside your browser, so nothing you type is uploaded or stored on a server. You can use the tool offline once the page has loaded, and the generated text exists only in your browser until you copy it.