Treat AI crawlers as three separate decisions: search visibility, model training, and fetches made on a user's behalf. OpenAI and Anthropic both publish a separate user agent for each purpose, so you can allow one and disallow another.
The crawlers, as documented
OpenAI lists three:
- OAI-SearchBot surfaces sites in ChatGPT search features. Opting out means your site will not appear in those answers, though it can still show as a navigational link.
- GPTBot crawls content that may be used to train generative models. Disallowing it signals that your content should not be used for training.
- ChatGPT-User fetches a page when a person asks ChatGPT or a custom GPT something. OpenAI says robots.txt rules may not apply to these user initiated visits.
Anthropic lists three as well:
- ClaudeBot collects content that may be used for training.
- Claude-SearchBot indexes content to improve search results.
- Claude-User retrieves pages when a user asks a question that needs them.
Anthropic says its bots honor robots.txt, support the Crawl-delay directive, and do not bypass CAPTCHAs.
A decision guide
Ask what you want from each purpose.
If you want to be findable in AI search, allow the search user agents. If you also want to opt out of training, disallow the training agents. The vendors say each setting is independent.
If you publish docs to drive adoption, being retrievable usually has value. If your content is paid or licensed, you may decide differently. That is a business choice, not a security one.
A sample file
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
This is an example of the mechanism, not a recommendation. Adjust it to your own policy.
Things that go wrong
Rules apply per host, so each subdomain needs its own file. Anthropic says to place the rule at the top level of each subdomain you want to block.
Do not block by IP address as a substitute. Anthropic notes this may not work reliably, because it stops the bot from reading your robots.txt at all.
Changes are not instant. OpenAI says search related robots.txt changes can take about 24 hours to take effect.
Robots.txt is a request, not access control. Anything truly private belongs behind authentication.
Verify, then monitor
Check your server logs for the user agent strings and compare against the vendor's published list where one exists. Anthropic points to a bots JSON list for this. A user agent string alone can be spoofed, so a verification step matters if the distinction is important to you.
Quick check
If I disallow GPTBot, do I vanish from ChatGPT search?
OpenAI says each setting is independent, so allowing OAI-SearchBot while disallowing GPTBot is a supported combination.
Does robots.txt stop a person from pasting my page into a chatbot?
No. It governs crawlers. A person can still copy text from a page they can read.
Sources
- OpenAI crawlers overview
- Anthropic crawler documentation
- Introduction to robots.txt, Google Search Central
Last reviewed: 19 October 2026.
