robots.txt Generator

Free robots.txt Generator

Create a properly formatted robots.txt file for your website. Control which pages search engine crawlers can access and specify your sitemap location. All processing happens in your browser.

What Is robots.txt?

A robots.txt file tells search engine crawlers which pages or files they can or can't request from your site. It's placed in the root directory of your website (e.g., https://example.com/robots.txt).

Common Directives

• User-agent: * — applies to all crawlers
• Disallow: /private/ — block a specific directory
• Allow: /public/ — explicitly allow access (useful when parent is disallowed)
• Crawl-delay: 10 — seconds between requests (respected by some bots)
• Sitemap: — tell crawlers where your XML sitemap is

Blocking AI Crawlers

Many website owners want to prevent AI training crawlers from scraping their content. Use the "Block AI crawlers" option to add rules for GPTBot (OpenAI), CCBot (Common Crawl), Google-Extended, and anthropic-ai.

Frequently Asked Questions

Where do I put robots.txt?

Upload it to the root of your domain, so it's accessible at https://yourdomain.com/robots.txt.

Does robots.txt block indexing?

No — it only prevents crawling. To prevent indexing, use a <meta name="robots" content="noindex"> tag or an X-Robots-Tag HTTP header.