Robots.txt & Sitemap Generator
A robots.txt file is the text file at your site's root that tells crawlers which paths they may fetch. This generator writes one, with an explicit AI-crawler policy, plus an XML sitemap.
How to use this
Building both files here is a four-step job with no account.
- 1Enter your website address.
- 2Choose whether to let AI assistants (like ChatGPT) read your site — we recommend yes.
- 3Copy the robots.txt text and save it as a file named robots.txt at the top level of your website.
- 4Copy the sitemap and save it as sitemap.xml — it’s the map search engines use to find all your pages.
robots.txt
robots.txt — serve at your domain root
User-agent: * Disallow: /admin Disallow: /api User-agent: GPTBot Allow: / User-agent: ClaudeBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Google-Extended Allow: / User-agent: CCBot Allow: /
XML sitemap
List your canonical URLs only. Serve the file at /sitemap.xml and declare it in robots.txt above.
Paste URLs — the sitemap appears here.
How to add your sitemap to robots.txt
Adding a sitemap to robots.txt is one line: the word Sitemap, a colon, and the sitemap's full URL, placed anywhere in the file.
Google's documentation gives the form as Sitemap: https://example.com/my_sitemap.xml. The URL must be fully qualified, with the protocol and host. The line is not tied to any user-agent group, and a file can list as many sitemaps as the site has.
- Put robots.txt in the top-level directory of the host: example.com/robots.txt, not a subfolder.
- Keep it under 500 KiB. Google ignores anything after that limit.
- Split a sitemap before it passes 50,000 URLs or 50 MB uncompressed, and list each file or a sitemap index.
- Give lastmod only when it is accurate. Google uses it when it is consistently verifiable and ignores priority and changefreq.
What happens when robots.txt is missing or failing
A missing robots.txt is harmless; a failing one is not. Google treats a 4xx response, except 429, as if no file existed, so everything may be crawled.
A 5xx response is the dangerous case. Google stops crawling the site while it retries, then falls back to the last copy it fetched. A server that errors on /robots.txt can stall the crawl of pages that are otherwise fine, so check that the file answers 200.
AI crawlers you can name in robots.txt
These are the user-agent tokens the four largest AI vendors document, what each one is for, and whether it follows robots.txt, as each vendor states it.
| User-agent token | Vendor | What it is for | Follows robots.txt |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Surfaces websites in ChatGPT's search features | Yes |
| GPTBot | OpenAI | Collects content to improve OpenAI's generative models | Yes |
| ChatGPT-User | OpenAI | Fetches a page when a ChatGPT user's action calls for it | May not apply (user-initiated) |
| Claude-SearchBot | Anthropic | Improves search result quality for Claude users | Yes |
| ClaudeBot | Anthropic | Collects web content for Anthropic's models | Yes |
| Claude-User | Anthropic | Fetches pages when a Claude user asks a question | Yes |
| PerplexityBot | Perplexity | Surfaces and links websites in Perplexity's search results | Yes |
| Perplexity-User | Perplexity | Visits a page to answer a user's question | Generally ignores it |
| Google-Extended | Controls use of content for training Gemini models | Yes — not a Search crawler |
Read from OpenAI, Anthropic, Perplexity and Google's own crawler documentation on 30 September 2026. Google says Google-Extended does not affect inclusion or ranking in Google Search.
Frequently asked questions
Where does the sitemap line go in robots.txt?+
Anywhere in the file. The Sitemap line is independent of the user-agent groups, so it can sit at the top or the bottom, and you can add one line per sitemap.
Does blocking GPTBot remove my site from ChatGPT search?+
No. OpenAI documents them as separate user agents: OAI-SearchBot surfaces sites in ChatGPT's search features, and GPTBot collects content for its models. Each can be allowed or blocked on its own line.
Does blocking Google-Extended hide my site from Google?+
No. Google states that Google-Extended does not affect a site's inclusion in Google Search and is not a ranking signal. It controls use of content for Gemini training.
Is robots.txt enough to keep a page out of search results?+
No. Google says a page disallowed in robots.txt can still be indexed if other sites link to it. To keep a page out, use a noindex meta tag or header on a page crawlers may fetch, or password-protect it.
Is this robots.txt and sitemap generator free?+
Yes. Both files are built in your browser, free, with no sign-up.
The rules that matter most in 2026
A robots.txt file now has to make two separate decisions that used to be one: what traditional search crawlers may fetch, and what AI crawlers may fetch. Treating them as a single group is the most common and most costly mistake in this file.
AI crawlers split further. Live search crawlers — OAI-SearchBot, PerplexityBot, Claude-SearchBot — fetch pages to answer questions right now, so blocking them removes you from AI answers today. Training crawlers such as GPTBot, ClaudeBot, CCBot and Google-Extended collect data for model training, and declining them is a legitimate content position that does not affect live retrieval.
Two failure modes account for most of the damage we see. A blanket rule inherited from a security plugin or CDN preset that disallows anything matching a bot pattern, catching the search crawlers nobody meant to block. And a robots.txt that allows everything while a web application firewall returns 403 to the same user agents, so the file says one thing and the server does another.
Check your access logs after any change: robots.txt is a request, and the only proof that a crawler is getting through is a 200 in your own logs.
Sitemaps deserve the same scepticism. A sitemap listing URLs that redirect, 404 or carry a canonical pointing elsewhere trains crawlers to trust it less, and honest lastmod values matter more than complete ones — a file claiming every page changed today is information-free.
Sources
- Google Search Central — How Google interprets the robots.txt specification
- Google Search Central — Build and submit a sitemap
- MDN Web Docs — robots.txt
- Google Search Central — Introduction to robots.txt
- Google — Common crawlers (Google-Extended)
- OpenAI — Overview of OpenAI crawlers
- Anthropic — Does Anthropic crawl data from the web?
- Perplexity — Perplexity crawlers
Last reviewed .