Most small business sites should not block AI crawlers that power search and citations, because blocking them removes you from the answers you want to appear in. Blocking training crawlers is a separate choice, and it depends on whether you mind your content being used to train models.
The two goals pull in opposite directions, which is why a blanket block or a blanket allow is usually wrong. Treat each bot according to what it does.
The bots, grouped by purpose
Company names and bot names change, so check each vendor’s documentation before you edit anything. At the time of writing, the main groups are:
Training crawlers. These collect content to train models. OpenAI’s GPTBot and Common Crawl’s CCBot are examples. Blocking them stops future training use of your pages, not past use.
Search and citation crawlers. These index pages so an AI product can cite them in answers. OpenAI’s OAI-SearchBot and Perplexity’s PerplexityBot are examples. Blocking them can keep your site out of those products’ answers.
User-triggered fetchers. These load a page because a person asked an AI tool about it. ChatGPT-User and Perplexity-User are examples. Both vendors say robots.txt rules may not apply to these, because a person requested the visit. Blocking them in robots.txt is not a reliable control.
Google’s control token. Google-Extended is a robots.txt token that lets you manage whether content Google crawls may be used to train future Gemini models. Google’s documentation also lists grounding, which means supplying Search index content to Gemini at prompt time. It says Google-Extended does not affect inclusion in Google Search and is not a ranking signal. That page does not mention AI Overviews or AI Mode, so do not assume the token controls them.
Each vendor documents its own bots:
- OpenAI: GPTBot, OAI-SearchBot and ChatGPT-User
- Perplexity: PerplexityBot and Perplexity-User
- Google: Google-Extended and other common crawlers
- Common Crawl: CCBot
| Purpose | Example user-agents | If you block |
|---|---|---|
| Training | GPTBot, CCBot, Google-Extended | Your pages are not used for future training |
| Search and citation | OAI-SearchBot, PerplexityBot | You may not be cited in that product |
| User-triggered | ChatGPT-User, Perplexity-User | Robots.txt may not stop these fetches |
| Normal search | Googlebot, Bingbot | You disappear from search results |
What robots.txt can and cannot do
Robots.txt is a request, not a lock. Well-behaved crawlers follow it; others ignore it. It stops nothing at the server level, and it does not remove content that has already been collected. For stronger control you need blocking at your CDN or firewall, which brings us to Cloudflare.
It also applies per path. You can block a crawler from one folder, such as a members area, and leave the rest open. That is useful if only part of your site is premium content.
What a robots.txt for AI bots looks like
A file that allows search and citation bots and blocks training bots might look like this. It is an example to understand the shape, not something to paste in unchecked:
User-agent: GPTBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: *
Allow: /Each group starts with the bot’s name and lists what it may or may not fetch. The final group is the default for any bot not named above.
Cloudflare content signals
Cloudflare introduced content signals as a way to state your preferences in robots.txt using a standard set of labels, such as whether content may be used for search, for AI answers or for training. You choose yes or no for each.
A typical stance for a business site that wants visibility:
search = yes
ai-input = yes
ai-train = noCheck Cloudflare’s content signals documentation for the exact syntax before copying anything, because the format is new and may change.
Content signals state a preference. Whether a crawler honours it is up to the crawler. If you want enforcement, Cloudflare also offers bot-blocking settings that act on the request itself. Check Cloudflare’s bot documentation for the current option names.
A decision guide
| Your goal | Search and citation bots | Training bots |
|---|---|---|
| Be cited in AI answers, protect content from training | Allow | Block |
| Be cited and accept training use | Allow | Allow |
| Keep content out of every AI product | Block | Block |
Most businesses that sell services belong in the first row. They gain from being named in answers, and they lose nothing by opting out of training.
Publishers whose content is the product may reasonably choose the third row, accepting lower visibility.
What blocking costs you
If you block search and citation bots, you will not appear in that product’s cited sources. If you block training bots, the effect is slower and harder to see: your content is not in future model training sets, which may mean the model knows less about your brand without browsing.
Do not block Googlebot or Bingbot to control AI use. Those bots feed normal search, and blocking them removes you from results.
How to make the change safely
- Back up your current robots.txt.
- Edit one group of bots at a time.
- Test the file in a robots.txt tester or by loading yoursite.com/robots.txt.
- Check server logs after a week to see which bots still arrive.
- Write the date and the change in a note so you can compare traffic later.
If you run WordPress, a robots.txt may be generated by a plugin or your host. Find out where yours comes from before editing, or your changes will be overwritten.
How to check whether bots obey
Your server logs or Cloudflare analytics show which user-agents request your pages. After you block GPTBot, you should see its requests stop or fail. If a bot keeps fetching pages it is told not to, you have learned it ignores robots.txt, and your options are a firewall rule or accepting it.
FAQ
Does blocking GPTBot remove me from ChatGPT answers?
GPTBot is the training crawler. Search results in ChatGPT use a separate bot. Check OpenAI’s documentation to confirm which bot does what.
Will blocking AI bots hurt my Google rankings?
Blocking Google-Extended is not documented as affecting rankings. Blocking Googlebot would.
Is an llms.txt file the same thing?
No. llms.txt is a proposal for a file that helps AI tools read your content, not an access control. Robots.txt is what governs crawling.
Can I change my mind later?
Yes. Robots.txt changes take effect as crawlers re-read the file. You cannot take back training that has already happened, but you can reopen access at any time.
