Robots.txt Generator
Generate a clean, RFC-compliant robots.txt file. Supports Allow/Disallow directives, XML sitemaps, crawl-delay, AI scraper controls (GPTBot, CCBot).
Crawler Permissions & Directives
🤖 AI Crawler & LLM Scraper Controls
Generated robots.txt
🔍 Live URL Path Evaluator
Matches disallow rule: Disallow: /admin/
⚖️ Decision Verdict: Production-Ready Robots.txt
Valid RFC 9309 structure. Search engines are invited to index public content while sensitive backend paths (/admin/, /private/) remain protected.
Robots Exclusion Telemetry
About this tool
How to use it
- Select a configuration template (Standard Public Website, AI Protected, Staging, WordPress, or E-Commerce).
- Define default access policies for User-agent: * and specify disallowed directory paths.
- Add allowed exceptions for public asset endpoints (e.g., /admin/public.css).
- Toggle AI scraper blocks (GPTBot, CCBot, Claude-Web, Google-Extended) as desired.
- Enter your XML sitemap URL and test individual URL paths in the live evaluator before downloading your robots.txt file.
Why it matters
The Robots Exclusion Standard (formalized in RFC 9309) is the web's universal protocol governing how automated web crawlers, search engine indexers, and LLM training spiders access a web server. An incorrect robots.txt file can accidentally de-index an entire commercial website from Google, while a properly configured one protects server bandwidth, prevents indexing of duplicate shopping cart URLs, and shields copyrighted content from AI training scrapes.
When to use it
Use this tool whenever launching a new web domain, migrating a staging website to production, optimizing crawl budget for high-volume sites, or preventing proprietary data scraping by generative AI bots.
Good to know
Placement Requirement: A robots.txt file must always be placed at the absolute root of your domain (e.g., https://example.com/robots.txt) and served as text/plain. Subdirectory placement (https://example.com/sub/robots.txt) will be ignored by search engine crawlers.
Common questions
Does robots.txt hide secret web pages?
No. Robots.txt is a publicly accessible file. Anyone can inspect it in their browser. Never put sensitive URLs or secret admin panel paths in robots.txt expecting privacy; use authentication and password protection instead.
What is the difference between noindex and Disallow?
`Disallow` in robots.txt stops spiders from crawling a page, but if external sites link to it, Google can still index the URL without reading its content. A `<meta name='robots' content='noindex'>` tag on the page itself ensures the page is completely dropped from search indices.
How do I block OpenAI from scraping my site for training?
Add `User-agent: GPTBot` followed by `Disallow: /` to your robots.txt. This explicitly forbids OpenAI's web crawler from scraping your site for model training.
Does Googlebot respect Crawl-delay?
No. Googlebot ignores the `Crawl-delay` directive. To manage Googlebot crawl rate, use Google Search Console. Other crawlers like Bingbot, Yandex, and Baidu do honor Crawl-delay.
What happens if a path matches both Allow and Disallow?
Under RFC 9309, if an Allow and Disallow rule both match a URL path, the most specific (longest) directive pattern takes precedence. If both rules have the exact same character length, the Allow directive wins.