Robots.txt Generator
Create robots.txt files with common presets
Configuration
Generated robots.txt
Common User-Agents
| Bot | User-Agent |
|---|---|
| All bots | * |
Googlebot | |
| Bing | Bingbot |
| DuckDuckGo | DuckDuckBot |
| ChatGPT | GPTBot |
| Claude | anthropic-ai |
| Common Crawl | CCBot |
About Robots.txt Generator
The Robots.txt Generator helps you create robots.txt files to control how search engines and crawlers access your website.
- User-agent: Specifies which bot the rules apply to (* means all)
- Allow: Paths that crawlers are permitted to access
- Disallow: Paths that crawlers should not access
- Crawl-delay: Seconds to wait between requests (not supported by all bots)
- Sitemap: Location of your XML sitemap