Robots.txt & Sitemap Generator

Control what crawlers may fetch — and decide explicitly whether AI crawlers are welcome (for GEO, they should be: no crawl, no citation).

robots.txt

Paths to block from crawling (one per line — leave empty to allow everything)Sitemap URL (recommended)

AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot)

Explicitly welcome them (recommended for GEO — you want to be cited)Block them (your content stays out of AI answers)Say nothing (they follow the * rules)

robots.txt — serve at your domain root

User-agent: *
Disallow: /admin
Disallow: /api

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: CCBot
Allow: /

XML sitemap

Your page URLs (one per line, full https:// addresses)

List your canonical URLs only. Serve the file at /sitemap.xml and declare it in robots.txt above.

Paste URLs — the sitemap appears here.